acceptodds
Under review as a conference paper at ICLR 2027

VQ-LLMTS: Vector Quantization-based Large Language Models for Time Series Forecasting and Understanding

30 views

Abstract

Large Language Models (LLMs) excel at reasoning but struggle with time-series data due to the modality gap between discrete text tokens and continuous numerical signals. In this paper, we propose VQ-LLMTS, a multimodal framework that addresses this gap by discretizing continuous time series into a specialized temporal vocabulary. Unlike many alignment methods that directly bridge text and time-series embeddings, we employ Residual Vector Quantization (RVQ) to decompose complex temporal patterns into a hierarchical, discrete latent space. This expands the LLM's vocabulary with learned temporal tokens, enabling unified cross-modal reasoning while preserving the original backbone and single-head autoregressive architecture. To capture diverse temporal patterns, we develop a generalized hybrid data generator for RVQ codebook training. We then align the temporal vocabulary with the language model using synthetic examples for general capabilities and time-series understanding, together with real-world univariate time series. VQ-LLMTS achieves strong time-series understanding performance and competitive short-horizon forecasting performance using qualitative descriptions derived from the target windows. On the CiK benchmark, VQ-LLMTS also achieves the lowest context-conditioned RCRPS among the compared methods. These results support the effectiveness of an explicit temporal vocabulary for LLM-based time-series understanding and forecasting.

Primary area:
learning on time series and dynamical systems
Keywords:
Time-Series Forecasting, Time-Series Understanding, Residual Vector Quantization, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dfgSOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VQ-LLMTS, a framework that integrates time series with LLMs by converting continuous time series into discrete tokens using RVQ, enabling time-series and textual tokens to be represented and generated within a shared autoregressive framework. The method consists of two stages: training an RVQ-based autoencoder to learn a discrete temporal vocabulary, followed by aligning the resulting time-series tokens with an LLM through a temporal projection module and parameter-efficient fine-tuning. The authors evaluate the framework on time-series understanding, short-horizon forecasting, the M4 benchmark, and context-aware forecasting, reporting competitive or improved results against several baselines.

Critical strengths
  1. Time series are represented as discrete tokens rather than directly using continuous embeddings.

  2. Residual Vector Quantization (RVQ) is employed to learn hierarchical representations of time-series patterns.

  3. The training framework consists of two stages. The first stage aims to learn a discrete temporal vocabulary that captures diverse temporal patterns, while the second stage performs modality alignment between time-series representations and natural language.

Critical weaknesses

W1. In the main forecasting experiments, VQ-LLMTS, LLM-SFT, VQ-LLMTS-Reg, and Time-LLM use textual descriptions derived from the future target windows, while the other baselines do not receive this information. These descriptions contain information about future trends, fluctuations, and extrema locations, which is generally unavailable from historical observations alone. Therefore, the performance differences across these groups cannot be attributed solely to the RVQ representation. Although the appendix includes experiments without future-derived context, they are mainly limited to ETTh1, which is insufficient to establish whether the reported advantages hold across other datasets.

W2. The main contribution of this paper is to convert continuous time series into discrete tokens using RVQ and integrate them into the LLM vocabulary. However, the comparison between LLM-SFT and VQ-LLMTS changes both the numerical encoding and the representation interface, making it difficult to isolate the contribution of the discrete representation itself. Moreover, VQ-LLMTS-Reg jointly updates the regression head, LoRA weights, and temporal projector, without clearly separating their individual contributions to the performance improvement.

W3. The paper compares against several established forecasting models, but many of these baselines are relatively old. Comparisons with more recent and competitive time-series forecasting models and LLM-based approaches would help provide stronger evidence of the proposed method's competitiveness.

Questions for authors

Q1. I have concerns about the availability of future-derived textual prompts in practical forecasting and the fairness of the corresponding experimental comparisons. In the main forecasting experiments, the model receives not only historical time series but also textual descriptions derived from the ground-truth future window, including its trend, patterns of change, fluctuations, and locations of extrema, and uses this information at test time. However, in conventional forecasting settings where the future is unknown, these descriptions cannot be obtained in advance using the generation procedure described in the paper. Therefore, these experiments evaluate conditional forecasting with future information. When comparing against baselines that do not receive equivalent prompts, it is difficult to attribute performance differences entirely to the model design. Although the authors disclose this setup and provide L0 results without future-derived prompts on ETTh1 in the appendix, this supplementary analysis on a single dataset remains insufficient to comprehensively assess the method’s performance under conventional forecasting conditions, while the main forecasting results and related discussion in the paper remain based on settings in which target-derived prompts are provided.

Q2. Could the authors provide additional experimental comparisons with recent related methods to more comprehensively assess the effectiveness of the proposed method and its performance relative to existing approaches?

Q3. Regarding the design of the RVQ module, could the authors further clarify its contributions beyond existing VQ/RVQ methods? Specifically, what substantive improvements does this work introduce to the encoder, residual quantization mechanism, or training objective to address the characteristics of time series? Compared with methods such as TOTEM, which use VQ-VAE to learn discrete time-series representations, what advantages do these design choices offer, and what experimental results support these advantages?

4 comments

Note: All shared supplementary tables (Tables R1–R5) are provided in our Global Response above.

W1 & Q1

Please see our response to the second reviewer (Reviewer LgVB, W2 & Q2) and Tables R3–R5. We added systematic experiments on practicality and fairness without ground-truth future text, including: (1) history-only L0 forecasting across all 14 datasets (8 standard + 6 M4 groups); (2) a two-stage pipeline predicting future windows via PatchTST from history alone to extract qualitative text; and (3) fixed-Oracle L4 text-noise ablations, Hourly/ILI non-monotonicity analyses, and real-world holiday forecasting with known operational priors.

W2 & Q2

  1. We compared against Continuous VAE to directly test continuous versus discrete representations on downstream forecasting (Table R2): under the identical Qwen2.5 backbone, regression interface, and L2 text input, despite Continuous VAE's lower raw reconstruction error (MAE 0.04589 vs. 0.1344), VQ-LLMTS-Reg beats Continuous VAE + Qwen2.5 on 13/14 datasets (8/8 on Matrix A; M4 weighted OWA 0.7460 vs. 0.8189), strictly isolating the contribution of discrete RVQ vocabulary alignment.

  2. We compared reconstruction fidelity and cost across discretization standards (Table R1): on 100k synthetic series, 2-layer RVQ reduces reconstruction MAE from 6.1252 (fixed-bin) and 0.5796 (Single-VQ) to 0.1344. Comparing generative and regression heads under L0 (Table R3), the regression head reduces MAE on smooth series (e.g., ETTm1 0.1109→0.0755, M4 Quarterly OWA 1.3839→0.8631), whereas the generative head excels on spiky/high-frequency series (e.g., ILI 2.0127→1.3391, M4 Hourly OWA 0.8769→0.6515).

W3 & Q3

We added comparisons with two recent baselines, i.e. TOTEM (purely unimodal time-series VQ-VAE) and ChatTime (fixed-bin LLM vocabulary expansion), across the same 14 datasets (Tables R2 & R3), finding:

  1. Beyond the benefits of two-layer RVQ for signal reconstruction, which reduces MAE and MSE by 76.8% and 92.7%, respectively, relative to single-VQ, we unify temporal and textual losses via Next-Token Prediction (NTP) during training, substantially strengthening cross-modal alignment and time-series understanding (Table 1: LLM-Judge 77.14, Regex Num 0.8821 vs. Chat-TS-14B 52.09 / 0.2711). Atop this aligned backbone, simply switching the output to a regression head flexibly adapts the model to accurate point forecasting.

  2. Compared with TOTEM, our unified, vocabulary-expanded LLM backbone supports both time-series language understanding and text-conditioned forecasting, while outperforming TOTEM on 7/8 Matrix A datasets under L0 history-only inputs and on all 8 datasets under L2 inputs. Compared with fixed-bin ChatTime, our method outperforms it in both waveform reconstruction (MAE 0.1344 vs. 6.1252) and downstream forecasting performance, outperforming ChatTime on all 8 Matrix A datasets and achieving a better M4 weighted OWA under L0 (0.8685 vs. 0.8782).

Reviewer_dfgSOpenReview ↗

Response to Authors

Thank you for the detailed response and additional experiments.

  1. Could the authors confirm whether a revised PDF has been uploaded? I find it difficult to identify the specific changes.

  2. Table R3 presents extensive experimental results. However, the method described in the manuscript uses LoRA. To my understanding, comparable LLM-based time-series forecasting methods generally keep the LLM frozen. Therefore, I view LoRA as an additional adaptation component, and this aspect has not been explained or discussed. Furthermore, the comparison between Single-VQ and RVQ still does not quantify the incremental contribution of RVQ over Single-VQ to downstream forecasting under the same LLM framework.

Since the authors have addressed several questions together, I am unsure whether my observations are correct. In particular, please correct me if my understanding regarding LoRA is mistaken.

  1. Although the additional experiments strengthen the evidence for the method’s effectiveness, I believe that, as presented in the current PDF, the paper’s substantive methodological contribution beyond existing VQ-based time-series representation methods and unified multimodal frameworks remains insufficiently clarified. Based on the evidence currently provided, I am maintaining my original rating of 2 for now.

Thank you for your comments. We clarify the three points below.

  1. Revised PDF and changes

Yes, we have prepared a revised PDF and will upload it together with this response. To make the changes easier to identify, the revised manuscript:

  • adds TOTEM, ChatTime, history-only L0, and PatchTST-predicted L2;
  • explicitly describes the LoRA, projector, generation-head, and regression-head adaptation settings;
  • presents the context definitions before the corresponding results;
  • includes the reconstruction comparisons, full L0–L4 study, text-perturbation results, and retail case in clearly identified tables.

2.1 LoRA and adaptation

You are right that the original backbone weights are frozen while LoRA adapters are trained. Thus, LoRA provides parameter-efficient adaptation without updating all original backbone parameters.

Stage II also updates the temporal projector and expanded generation head. VQ-LLMTS-Reg further adapts LoRA and the temporal projector together with its regression head. These are deliberate choices for adapting the temporal vocabulary and forecasting interface, rather than leaving the entire model unchanged. We have now stated them explicitly in Section 4.3 and Appendix A.1.1.

LoRA itself is not our claimed novelty. Our contribution is the temporal-vocabulary representation and its integration with the LLM.

2.2 RVQ and its downstream benefit

Single-VQ and RVQ both use discrete temporal representations. RVQ additionally models the residual left by the first codebook, preserving more local variations and fine-grained temporal structure. Downstream learning therefore receives less lossy discrete evidence about the trajectory, which provides a direct mechanism for improved forecasting.

The reconstruction results support this mechanism:

  • RVQ reduces reconstruction MAE from 0.5796 to 0.1344, a 76.8% reduction.
  • Truncating the trained RVQ to its first codebook gives MAE 0.6632, much worse than the full two-stage result. This shows that the residual stage supplies useful additional information rather than being redundant.

Thus, RVQ is beneficial not merely because it reconstructs better, but because it preserves more task-relevant temporal information for discrete forecasting.

  1. Methodological contribution

Our contribution is the integration of learned hierarchical temporal codes into the LLM’s input and output vocabulary, with joint text–time-series alignment and a shared generation head for textual descriptions and temporal codes. A separately adapted regression variant provides the numerical forecasting interface.

This differs from:

  • TOTEM, which does not integrate its VQ representations into an LLM text vocabulary;
  • ChatTime, which uses fixed scalar discretization rather than learned residual temporal codes.

The revised introduction and related-work section now make these distinctions explicit.

Thank you again for helping us clarify the method and its contribution.

Reviewer_dfgSOpenReview ↗

Response to Authors

Thank you for your response. Considering both the paper’s core contributions and experimental validation, I will maintain my current rating for now. I will participate in the upcoming discussion and reconsider my assessment in light of the other reviewers’ views.

Reviewer_LgVBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper constructs a temporal vocabulary through residual vector quantization and integrates it into a language model for joint numerical and textual generation. The model outperforms the compared methods on synthetic understanding tasks using the same generator as training, exceeds some baselines in conditional forecasting with target-derived textual descriptions, and achieves the lowest RCRPS among the reported CiK comparisons.

Critical strengths
  1. The method achieves competitive performance on the evaluated time-series understanding and conditional forecasting tasks.
  2. The paper is clearly written, with an easy-to-follow organization of the method and experiments.
Critical weaknesses
  1. The contribution of the proposed method could be further clarified. The framework combines residual vector quantization, vocabulary expansion, and autoregressive training. TOTEM explores VQ-based temporal representations, while ChatTime expands a pretrained LLM’s vocabulary to jointly process text and time series. Clarifying the differences and comparing RVQ with single-codebook and fixed-bin alternatives under matched training conditions would help identify which aspects account for the reported gains.

  2. Experiments use descriptions derived from ground-truth future windows, which would generally be unavailable at prediction time. Beyond the history-only ETTh1 comparison in Appendix A.7.2, broader history-only evaluation would help establish practical applicability. Alternatively, Chronos or Timer could predict future trajectories from historical observations, and the same rules could extract descriptions from these predictions. Comparing this pipeline with the forecaster itself and input-matched baselines would clarify the benefit without access to true future trends.

Questions for authors
  1. Could you clarify the specific contribution beyond ChatTime’s vocabulary expansion and TOTEM’s VQ-based representations? In particular, how does RVQ compare with single-codebook VQ and fixed-bin tokenization under the same backbone, training data, and budget on both understanding and forecasting? This would help distinguish the contribution of the residual quantization design from that of the alignment data and training procedure.

  2. Are the forecasting gains retained without target-derived descriptions across the other datasets and M4, beyond the ETTh1 study? Alternatively, could Chronos or Timer generate future predictions using only historical observations, from which the qualitative descriptions are extracted? Comparing the resulting pipeline with the forecaster alone and baselines receiving the same descriptions would help assess practical utility; it would also be useful to clarify the performance gap relative to true-future descriptions.

3 comments

Note: All shared supplementary tables (Tables R1–R5) are provided in our Global Response above.

W1 & Q1

TOTEM applies Single-VQ only to unimodal series, whereas ChatTime uses min-max fixed bins. VQ-LLMTS combines 2-layer RVQ with unified NTP alignment in an expanded vocabulary, supporting both time-series understanding and forecasting.

Comparing fixed-bin, Single-VQ, 2-layer RVQ, and Continuous VAE on the same 100k train / 10k test series (500 epochs, 17,729 trainable params; Table R1), 2-layer RVQ cuts reconstruction MAE from 6.1252 and 0.5796 to 0.1344 (-76.8%) and MSE from 0.6486 to 0.04715 (-92.7%). Masking the second codebook yields MAE 0.6632 (+0.08 GiB peak memory), which confirm the residual layer's role. Continuous VAE has lower reconstruction error at 0.04589, but continuous-discrete misalignment adversely affects downstream accuracy: under Qwen2.5 and L2 text, VQ-LLMTS-Reg outperforms Continuous VAE on 13/14 datasets (M4 OWA 0.7460 vs. 0.8189; Table R2). Under history-only L0 (Table R3), it also beats ChatTime on all 8 standard datasets plus M4 OWA (0.8685 vs. 0.8782) and beats TOTEM on 7/8 standard datasets.

W2 & Q2

We evaluated two practical regimes across all 14 benchmarks and an industrial deployment.

Regime 1: Scenarios with known operational qualitative priors.

Beyond public CiK (Table 4, best RCRPS 0.2730 across 355 tasks), retail supply chains routinely have known priors, such as transferring holiday shapes from mature cities to new ones or encoding planned promotions, closures, and listings. In a 24-day 2026 Spring and Lantern Festival test on a large-scale instant-retail platform with 35-day lookback and peak or store-event prompts, VQ-LLMTS raises 1 - WMAPE accuracy from 71.05% to 77.52% (+6.47 pp) over the production Transformer with holiday covariates (Table R6):

Table R6. Holiday Retail Forecasting (24d, 1 - WMAPE)

Model (1 - WMAPE)Overall (24d)Spring Fest.Lantern Fest.
Prod. Transformer (w/ cov.)71.05%62.20%59.58%
VQ-LLMTS (Ours)77.52%74.52%66.33%
Gain+6.47 pp+12.32 pp+6.75 pp

Regime 2: Conventional forecasting without prior future text (Tables R3–R5).

  1. History-Only L0 (14 Datasets): Without future text description, VQ-LLMTS-Reg (L0) beats the non-VQ LLM on 11/14 datasets, or 13/14 taking the better of the generative and regression heads per dataset, with only M4 Daily unimproved. Reg reaches an M4 weighted OWA of 0.8685 vs. 1.0840 without VQ, beating 14/15 paper baselines including PatchTST (0.9794), Time-LLM (1.0922), S2IP-LLM (0.8937), and Timer (2.1005), while matching Chronos (0.8683). On the 8 standard datasets, VQ-LLMTS-Reg leads all baselines on ETTh2 MSE, ranks 2nd to Chronos on Weather MAE, Traffic MAE, and Traffic MSE, ranks 4th/3rd on Electricity MAE/MSE, sits mid-pack on ETTh1/ETTm1/ETTm2, and trails on ILI (Table R4).

  2. Predicted-Context Pipeline (PatchTST, Table R7): Forecasting the future window via PatchTST to extract L1-L4 text yields non-monotonic gains across granularities, with L2 best on M4 weighted OWA. VQ-LLMTS-Reg + PatchTST (L2) lowers M4 OWA from 0.9794 for PatchTST and 0.8685 for L0 to 0.8485, beating all 15 baselines (best baseline Chronos: 0.8683). L2 beats L0 on M4 Quarterly, Monthly, Weekly, Daily, Hourly, ETTh1, and ILI. On the 8 standard datasets, only ETTh1 and ILI improve in both MAE and MSE, and M4 Yearly, Daily, and Hourly still trail the best baselines. Predicted text thus helps selectively rather than universally over L0.

  3. Oracle Context and Perturbations (Table R5): Oracle priors use target-window information to measure how much exact future context helps forecasting, complementing the history-only and history-predicted evaluations. Across the same 14 evaluation groups, L4 outperforms L2 on 12 groups for Reg and 10 groups for VQ-LLMTS. Within each level, the model weights, history windows, and numerical targets stay fixed while only the evaluation text changes. Trend flip increases error on 13/14 groups for Reg and all 14 for VQ-LLMTS; combined noise increases error on all 14 for both. M4 Hourly combines a strong diurnal pattern with a weak linear trend: Reg stays nearly constant (2.5761 vs. 2.5760/2.5768/2.5768), whereas VQ-LLMTS changes from 0.6170 to 0.6606/0.6593, showing that the two heads respond differently to the same perturbations. On ILI, spikes dominate the segment and extrema cues: Reg goes from 1.9110 at L2 to 2.1045 at exact L4, illustrating brittle fine-grained cues. In short, L2 is a reliable default, while L4 is most beneficial when its detailed cues faithfully capture the trajectory. This supports using context detail according to its accuracy rather than treating more detail as inherently better.

Table R7. VQ-LLMTS-Reg x PatchTST Predicted Context L0-L4 (Matrix A: MAE/MSE; M4: sMAPE/MASE/OWA)

DatasetL0 (Hist-only)PatchTST L1PatchTST L2PatchTST L3PatchTST L4
ETTh10.1354 / 0.03240.1328 / 0.03070.1330 / 0.03140.1345 / 0.03240.1309 / 0.0303
ETTh20.1873 / 0.06310.1881 / 0.06480.1954 / 0.07020.1992 / 0.07140.1915 / 0.0670
ETTm10.0755 / 0.01130.0765 / 0.01150.0768 / 0.01160.0764 / 0.01140.0757 / 0.0114
ETTm20.1014 / 0.02300.1009 / 0.02250.1014 / 0.02310.1024 / 0.02360.1003 / 0.0225
Electricity0.2094 / 0.07920.2129 / 0.08170.2120 / 0.08180.2101 / 0.08000.2156 / 0.0845
Weather0.0136 / 0.00050.0139 / 0.00050.0137 / 0.00050.0139 / 0.00060.0140 / 0.0005
Traffic0.1665 / 0.10740.1688 / 0.10810.1694 / 0.10940.1682 / 0.11000.1705 / 0.1098
ILI2.0127 / 7.39872.0209 / 7.11651.9587 / 7.31731.9650 / 7.13752.0570 / 7.4634
M4 Yearly15.0238 / 3.3347 / 0.879116.8840 / 3.7827 / 0.992316.3891 / 3.6745 / 0.963616.4108 / 3.6208 / 0.957617.4049 / 3.8706 / 1.0194
M4 Quarterly9.9979 / 1.1220 / 0.863110.2969 / 1.1472 / 0.88599.8748 / 1.0942 / 0.84749.4666 / 1.0520 / 0.81359.9585 / 1.0979 / 0.8525
M4 Monthly12.1906 / 0.8857 / 0.839212.1478 / 0.8640 / 0.827311.2674 / 0.8056 / 0.769412.2781 / 0.8803 / 0.839612.7605 / 0.9029 / 0.8668
M4 Weekly7.4203 / 1.8115 / 0.73017.5761 / 2.1053 / 0.79317.1949 / 1.7264 / 0.70237.4976 / 1.9179 / 0.75408.3396 / 2.2299 / 0.8567
M4 Daily3.5834 / 3.9107 / 1.18502.9505 / 3.1224 / 0.96073.4735 / 3.7280 / 1.13903.3328 / 3.6496 / 1.10413.0346 / 3.2101 / 0.9879
M4 Hourly14.9595 / 2.4081 / 0.876956.0264 / 42.4586 / 11.530615.4832 / 2.2148 / 0.839114.6706 / 2.1179 / 0.799894.1919 / 105.5229 / 27.8049
M4 Weighted Avg. (OWA)0.86850.92910.84850.87121.0151

Dear Reviewer LgVB,

Thank you again for your constructive review. Your suggestions directly shaped this revision.

We clarified our contribution beyond TOTEM and ChatTime, explained how residual quantization preserves temporal information for downstream forecasting, and added history-only experiments together with the PatchTST-based predicted-context pipeline you suggested.

We would be grateful if you could take another look at the revised manuscript and share any remaining questions or concerns before the discussion period ends. We would be glad to continue the discussion.

Best regards, Authors

Reviewer_JCz5OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VQ-LLMTS, a framework that integrates time series with LLMs by converting continuous time series into discrete tokens using RVQ, enabling time-series and textual tokens to be represented and generated within a shared autoregressive framework. The method consists of two stages: training an RVQ-based autoencoder to learn a discrete temporal vocabulary, followed by aligning the resulting time-series tokens with an LLM through a temporal projection module and parameter-efficient fine-tuning. The authors evaluate the framework on time-series understanding, short-horizon forecasting, the M4 benchmark, and context-aware forecasting, reporting competitive or improved results against several baselines.

Critical strengths
  1. The paper proposes an approach to integrating time series with LLMs by converting continuous temporal signals into discrete tokens and incorporating them into a shared vocabulary with textual tokens.
  2. The paper examines both the autoregressive VQ-LLMTS framework and the regression-oriented VQ-LLMTS-Reg variant, considering two prediction paradigms within the proposed framework.
Critical weaknesses
  1. Although the paper identifies the modality gap between time series and language, it does not sufficiently explain why RVQ-based discretization is necessary or preferable to approaches using continuous representations.
  2. The framework combines RVQ, temporal projection, cross-modal alignment, and LLM fine-tuning, but the paper does not clearly distinguish its methodological contributions from existing work on time-series tokenization and LLM-based forecasting. A more explicit articulation of the paper’s key contributions and a clearer explanation of the unique benefits of RVQ within the proposed framework are needed to establish its novelty.
  3. Although the comparison with direct SFT of the same LLM is useful, it does not fully disentangle the effects of RVQ from those of temporal projection, cross-modal alignment, and differences in training objectives. More controlled ablations, particularly comparisons between continuous and quantized representations and between standard VQ and RVQ, are needed to determine whether the proposed discrete temporal vocabulary is responsible for the reported gains. This limits the empirical support for the paper’s central methodological claim.
Questions for authors
  1. What specific limitations of continuous time-series representations does the proposed RVQ-based discretization address, and why is it expected to be more suitable for LLM-based time-series understanding and forecasting?
  2. Could the authors explicitly identify the main methodological contributions of VQ-LLMTS and clarify how they differ from existing approaches to time-series tokenization and LLM-based forecasting?
  3. Can the authors provide controlled ablation results comparing continuous and quantized time-series representations, as well as standard VQ and RVQ, while keeping other components and training settings as consistent as possible? Such comparisons would help determine whether the reported gains can be attributed specifically to RVQ-based discretization rather than to temporal projection, cross-modal alignment, or differences in training objectives.
3 comments

Note: All shared supplementary tables (Tables R1–R5) are provided in our Global Response above.

W1 & Q1

  1. Continuous representations face two bottlenecks in joint text–series modeling: (i) Continuous projection requires a separate forecast head (Time-LLM [13], GPT4TS [49], S²IP-LLM [29], AutoTimes [23]). Because time-series values are not part of the LLM’s vocabulary, the model cannot generate time series and natural-language explanations by a single causal-LM pass, nor can it be precisely controlled by preceding text (Tables 1–4; e.g., on CiK in Table 4, adding text degrades Time-LLM RCRPS from 0.5839 to 0.6577, whereas VQ-LLMTS achieves the best 0.2730 with 0.0259 constraint error). (ii) The effectiveness of discrete vocabularies for unified autoregressive generation has been validated in audio and video generation [45, 6, 2, 35]. In time series forecasting, Chronos [1] validated the use of quantized tokens with cross-entropy training, whereas TOTEM [38] does not integrate vector quantization into an LLM vocabulary.

  2. Controlled comparisons among fixed-bin quantization, single-VQ, two-layer RVQ, and a deterministic continuous VAE under identical settings (Tables R1 & R2) show that a lower raw signal-reconstruction error does not necessarily translate into better forecasting performance. Although the continuous VAE achieves a lower reconstruction MAE than RVQ (0.04589 vs. 0.1344), VQ-LLMTS-Reg with the same Qwen2.5 backbone and L2 input outperforms Continuous VAE with Qwen2.5 backbone on 13 of 14 datasets, including all 8 Matrix A datasets. Generative VQ-LLMTS also outperforms the continuous-VAE counterpart on 9 of 14 datasets. These results suggest that continuous latents face cross-modal alignment challenges, whereas an expanded discrete vocabulary provides more effective alignment for downstream forecasting.

W2 & Q2

We will clarify our contribution boundaries in Sections 1–2 (and add discussions of TOTEM and ChatTime); our core contributions comprise three parts:

  1. Two-Layer Hierarchical RVQ (with Boundary Padding): At identical parameter count (17,729), RVQ reduces reconstruction MAE by 97.8% vs. fixed-bin (ChatTime) and 76.8% vs. Single-VQ (TOTEM) (Table R1), removing multi-scale quantization bottlenecks.

  2. Joint NTP Alignment of Text and Time Series in an Expanded Unified Vocabulary, supporting both understanding and forecasting:

2.1 Strong Time-Series Understanding: Unimodal VQ (TOTEM) cannot output text, whereas VQ-LLMTS achieves BLEU 0.8879, Regex Num 0.8821 (far surpassing GPT-5.4's 0.2568, DeepSeek-V3's 0.2657, and Chat-TS-14B's 0.2711), and LLM-Judge 77.14 (vs. runner-up Chat-TS-14B at 52.09).

2.2 Flexible Dual-Mode Forecasting: After cross-modal fusion in the discrete vocabulary, the model supports both autoregressive generation (suited to spiky/multimodal regimes like ILI, Traffic, M4 Hourly, CiK) and lightweight switching to a regression head (VQ-LLMTS-Reg). Our main distinction lies in the unified expanded vocabulary, time-series language understanding, and text-conditioned forecasting; for forecasting, Reg outperforms Continuous VAE and ChatTime on 13/14 datasets and beats TOTEM on 7/8 Matrix A datasets under history-only L0 (Tables R2 & R3).

W3 & Q3

To isolate the RVQ vocabulary from projection layers and alignment objectives, we added two ablation studies:

  1. Representation-Level Ablation (Table R1): Fixing the encoder, 100k training set, 500 epochs, and trainable parameters (17,729), 2-layer RVQ significantly outperforms fixed-bin and Single-VQ; masking the 2nd residual codebook degrades MAE from 0.1344 to 0.6632, directly isolating residual quantization.

  2. Same-Backbone Downstream Ablation (Tables R2 & R3): Fixing Qwen2.5, the regression head, and L2 input, RVQ beats Continuous VAE on 13/14 datasets despite Continuous VAE's lower reconstruction error. Under history-only L0 without future text (Table R3), VQ-LLMTS-Reg beats the non-VQ LLM on 11/14 datasets—or 13/14 when taking the better of generative and regression heads per dataset (Reg M4 weighted OWA 0.8685 vs. 1.0840)—and beats ChatTime on 8/8 Matrix A datasets plus M4 OWA (0.8782), supporting the contribution of RVQ discrete vocabulary alignment.

Thank you for your response. However, my primary concern remains the technical contribution of this paper. The differences highlighted by the authors can not excite me. Therefore, I believe my current score remains appropriate.

Thank you for your follow-up. We respectfully believe that VQ-LLMTS’s contribution goes beyond combining RVQ with LLM fine-tuning.

Our central contribution is a learned, hierarchical temporal vocabulary that brings text and time series into the same expanded vocabulary space. Sharing an embedding dimension is not equivalent to sharing a generative vocabulary. After next-token alignment, the framework naturally supports either a language-model head for joint text–temporal generation or a regression head for numerical forecasting through task-specific adaptation. The contribution therefore lies in the unified representation and its flexibility, rather than a particular output head.

The evidence spans both understanding and forecasting. The autoregressive model achieves strong performance on understanding metrics and the lowest context-conditioned RCRPS among the compared methods on CiK. On standard forecasting datasets, the regression variant outperforms the compared continuous-VAE model under matched backbone and target-derived contextual inputs, despite the latter’s better reconstruction fidelity.

Given computational constraints, our backbone adaptation uses LoRA rather than full-parameter fine-tuning. We believe further scaling of model capacity, training data, and adaptation could unlock substantial potential for joint time-series–text understanding and forecasting.

We therefore hope the work is assessed as a contribution to joint language–time-series modeling, rather than merely an incremental quantization modification, and respectfully invite you to reconsider your assessment.

Comments on the paper

5 comments

We thank all three reviewers (JCz5, LgVB, and dfgS) for their constructive feedback. Four shared questions emerged:

  1. Why is RVQ discrete vocabulary alignment needed over continuous representations, Single-VQ, or fixed-binning? (JCz5 W1–W3, LgVB W1/Q1, dfgS W2/Q3)

  2. How does VQ-LLMTS perform without ground-truth future text? (LgVB W2/Q2)

  3. How do text granularity and noise affect forecasting? (dfgS W1/Q1)

  4. How competitive is it against recent discrete and vocabulary-expanded baselines (TOTEM, ChatTime, Chronos, Timer, S²IP-LLM)? (LgVB W1, dfgS W3/Q2)

We completed five supplementary experiment suites, with downstream evaluations covering all 14 benchmarks (8 standard datasets + 6 M4 groups; Tables R1–R5):

1. Reconstruction & Cross-Modal Alignment (Tables R1 & R2)

On 100k synthetic series (500 epochs), 2-layer RVQ cuts reconstruction MAE from 6.1252 (fixed-bin) and 0.5796 (Single-VQ) to 0.1344 (-76.8% MAE / -92.7% MSE vs. Single-VQ), with no notable overhead in inference cost or speed (training time 24.63 vs. 24.57 min; throughput 153,584 seq/s). Although Continuous VAE has lower raw reconstruction error (0.04589), VQ-LLMTS-Reg outperforms Continuous VAE + Qwen2.5 on 13/14 datasets under the same Qwen 2.5 backbone and L2 text richness (Table R2), demonstrating that cross-modal alignment in an extended discrete vocabulary drives downstream gains.

2. Forecasting without Future Text Descriptions (Tables R3, R4)

Under history-only L0 setting, VQ-LLMTS-Reg achieves a weighted OWA of 0.8685 on M4 dataset, beating 14/15 prior-work benchmarks and matching Chronos of 0.8683 (Table R4-part2). Across the 8 Matrix A datasets ( Table R4-part1a &b), it outperforms all prior-work benchmarks on ETTh2 in terms of MSE, ranks second only to Chronos on Weather and Traffic in MAE, as well as on Traffic in MSE, ranks 4th and 3rd on Electricity in MAE and MSE respectively. It performs mid-pack on ETTh1/ETTm1/ETTm2, and lags behind on ILI.

Suggested by Reviewer LgVB, we can use history forecasts to generate text first (we use PatchTST ), and then feed into our VQ model. As shown in Table R4-part2, on the M4 dataset, using the generated L2 text descriptions reduces the M4 OWA from 0.8685 to 0.8485, achieving the best result among all 15 prior-work benchmarks. However, across the Matrix A datasets, improvements in both MAE and MSE are observed only on ETTh1 and ILI.

Thus, VQ-LLMTS-Reg is competitive without future text and achieves the best leakage-free M4 OWA. However, predicted text provides a selective enhancement rather than a universal improvement over the L0 setting.

3. Text Granularity & Noise Sensitivity (Table R5)

Our experiments show that Finer-grained text is not necessarily better and its effectiveness depends on the accuracy of the added details, the predictability of the series, and the model’s ability to exploit the text. With oracle text, exact L4 descriptions improve over L2 descriptions on 12 of 14 datasets for VQ-LLMTS-Reg, but L2 outperforms L4 on M4 Hourly (OWA: 1.2106 vs. 2.5761) and ILI (MAE: 1.9110 vs. 2.1045). However, reversing the trend descriptions (i.e., the Flip row) degrades VQ-LLMTS-Reg’s performance on 13 of the 14 datasets, while introducing all three types of noise degrades performance on all 14 datasets (the Comb. row), with particularly pronounced degradation at the L4 level. For example, ETTh1 MAE rises from 0.090262 to 0.1634 on L4 description but reversing the trend descriptions. These results suggest that L2 is a robust default granularity, whereas L4 is beneficial only when trend, extrema, and segment-level cues are sufficiently accurate.

Furthermore, in a real business case with genuine prior information ( 24-day Spring Festival sales patterns of a large-scale instant retailer) VQ-LLMTS improves 1−WMAPE accuracy from 71.05% to 77.52%, which we can include in the final version if needed.

4. Competitiveness with Other Discrete/Expanded-Vocabulary Baselines (Tables R2, R3 & R4)

(i) Compared with TOTEM, VQ-LLMTS uniquely supports natural-language understanding and text-constrained forecasting (CiK), while beating TOTEM on 7/8 Matrix A datasets under L0 (8/8 under L2) in Table R2.

(ii) Compared with fixed-bin ChatTime, 2-layer RVQ avoids severe binning distortion (MAE 0.1344 vs. 6.1252), beating ChatTime on 8/8 Matrix A datasets and M4 OWA under L0 (0.8685 vs. 0.8782) and on 13/14 datasets under L2 (0.7460 vs. 0.8782) in Table R2.

(iii) Compared with Chronos, Timer, and S²IP-LLM, L0 VQ-LLMTS-Reg beats Timer on 14/14 datasets, beats S²IP-LLM on M4 OWA (0.8685 vs. 0.8937), matches Chronos (0.8683), and outperforms Chronos on M4 with PatchTST generated L2 text (0.8485) in Table R4 and on CiK (RCRPS 0.2730 vs. 0.3906).

More broadly, our unified vocabulary-expanded LLM backbone simultaneously supports time-series understanding and text-conditioned forecasting, combining flexibility and precision.

Table R1. Reconstruction Fidelity and Efficiency Cost (100k Train / 10k Test, Length=100, 500 Epochs, Latent Dim=64, K=512)

MetricFixed-binSingle-VQ (1-Layer)RVQ (2-Layer, Ours)Continuous VAE
Recon. MAE ↓6.12520.5796 (Stage-1 only: 0.6632)0.13440.04589
Recon. MSE ↓71.28470.64860.047150.003935
Bitrate9 bits/step9 bits/step18 bits/stepContinuous latent
Trainable Params17,72917,72917,72925,985
Codebook Buffer32,76832,76865,5360
Total Params + Buffer50,49750,49783,26525,985
Peak Inference Memory0.751 GiB0.752 GiB0.833 GiB0.210 GiB
Training Time24.73 min24.57 min24.63 min24.52 min
Inference Throughput274,896 /s226,861 /s153,584 /s665,651 /s

Table R2. Continuous VAE + Qwen2.5 vs. TOTEM and ChatTime (Matrix A: MAE/MSE; M4: sMAPE/MASE/OWA)

DatasetContinuous VAE + Qwen2.5 (L2)VQ-LLMTS-Reg (L2, Ours)VQ-LLMTS (L2, Ours)TOTEM (Hist-only)ChatTime (L2)
ETTh10.1417 / 0.03230.1087 / 0.02010.1180 / 0.02430.1364 / 0.03170.1385 / 0.0339
ETTh20.2301 / 0.08450.1608 / 0.04600.1805 / 0.06290.2095 / 0.07560.2601 / 0.1250
ETTm10.0829 / 0.01230.0616 / 0.00710.0941 / 0.01260.0819 / 0.01270.0847 / 0.0144
ETTm20.1683 / 0.04870.0848 / 0.01490.0986 / 0.02190.1260 / 0.03110.2037 / 0.1033
Electricity0.2297 / 0.09810.2046 / 0.07640.2298 / 0.10590.2143 / 0.08540.3111 / 0.2186
Weather0.0156 / 0.00060.0103 / 0.00030.0133 / 0.00040.0187 / 0.00080.0145 / 0.0006
Traffic0.2052 / 0.13820.1663 / 0.10310.1342 / 0.11940.2281 / 0.14900.3704 / 0.4269
ILI2.3246 / 8.86591.9110 / 6.75871.2653 / 4.38911.6937 / 6.70312.1493 / 10.9957
M4 Yearly14.4580 / 3.2518 / 0.851312.5648 / 2.7861 / 0.734913.1923 / 2.7331 / 0.747513.8814 / 3.3300 / 0.843019.0484 / 4.6653 / 1.1686
M4 Quarterly10.0798 / 1.0625 / 0.84518.9807 / 0.9701 / 0.761511.2276 / 1.1381 / 0.92477.8981 / 0.9216 / 0.69479.5775 / 1.2233 / 0.8811
M4 Monthly11.2875 / 0.7688 / 0.752410.9303 / 0.7676 / 0.739716.9670 / 1.2380 / 1.17038.9814 / 0.6440 / 0.614210.4111 / 0.8069 / 0.7410
M4 Weekly6.9453 / 1.7059 / 0.68536.6688 / 1.7700 / 0.68269.2290 / 2.3424 / 0.92546.8395 / 1.7502 / 0.68798.0307 / 2.0194 / 0.8011
M4 Daily3.9172 / 4.0716 / 1.26422.3370 / 2.3965 / 0.74923.7289 / 4.1916 / 1.25152.4692 / 2.6177 / 0.80472.6386 / 2.8574 / 0.8692
M4 Hourly15.3830 / 1.9862 / 0.780916.6110 / 3.6346 / 1.210614.1839 / 1.0451 / 0.604015.3606 / 1.2315 / 0.594314.2013 / 1.4722 / 0.6317
M4 Weighted Avg.11.4167 / 1.5584 / 0.818910.4833 / 1.3648 / 0.746014.1225 / 1.6859 / 1.01439.5919 / 1.4182 / 0.694411.8762 / 1.8880 / 0.8782

Table R3. L0 History-Only vs. Oracle (Matrix A: MAE/MSE; M4: OWA)

DatasetLLM (PT, no VQ)VQ-LLMTS (L0)VQ-LLMTS-Reg (L0 Hist-only)VQ-LLMTS-Reg (Oracle L2 / L4)
ETTh10.1593 / 0.04520.1324 / 0.03230.1354 / 0.03240.1087 / 0.0903
ETTh20.2925 / 0.16220.2043 / 0.08420.1873 / 0.06310.1608 / 0.1424
ETTm10.1134 / 0.02590.1109 / 0.01750.0755 / 0.01130.0616 / 0.0479
ETTm20.3141 / 0.22040.4310 / 0.43740.1014 / 0.02300.0848 / 0.0693
Electricity0.3204 / 0.29310.2318 / 0.10720.2094 / 0.07920.2046 / 0.2012
Weather0.0189 / 0.00090.0162 / 0.00060.0136 / 0.00050.0103 / 0.0079
Traffic0.3222 / 0.43470.1334 / 0.13810.1665 / 0.10740.1663 / 0.1551
ILI1.6674 / 5.99061.3391 / 5.03002.0127 / 7.39871.9110 / 2.1045
M4 Yearly (OWA)1.51251.02020.87910.7349 / 0.7004
M4 Quarterly (OWA)0.87191.38390.86310.7615 / 0.6356
M4 Monthly (OWA)0.98771.87910.83920.7397 / 0.6424
M4 Weekly (OWA)1.15181.28750.73010.6826 / 0.5327
M4 Daily (OWA)1.08402.00341.18500.7492 / 0.7478
M4 Hourly (OWA)0.68390.65150.87691.2106 / 2.5761
M4 Weighted Avg. OWA1.08401.56080.86850.7460 / 0.6662

Table R4. Leakage-Free Results vs. Paper Benchmarks (Future text: - = None; Lower is Better)

Part 1a: ETT (MAE/MSE)

MethodFuture textETTh1ETTh2ETTm1ETTm2
Time-LLMOracle L20.1228/0.02580.2009/0.07170.0713/0.00990.0987/0.0209
GPT4TS-0.1158/0.02350.1814/0.06460.0676/0.01020.0784/0.0153
S2IP-LLM-0.1165/0.02370.1905/0.06670.0675/0.00910.0757/0.0149
Chronos-0.1464/0.03750.1994/0.07820.0921/0.01680.0885/0.0223
Timer-0.1692/0.04650.3234/0.17110.1204/0.02470.2823/0.1305
DLinear-0.1209/0.02540.1867/0.06460.0799/0.01270.0886/0.0191
PatchTST-0.1187/0.02470.1855/0.06430.0680/0.00920.0828/0.0167
TimesNet-0.1216/0.02570.1876/0.06380.0693/0.00950.0823/0.0181
FEDformer-0.1323/0.02870.2036/0.07070.0733/0.01020.0988/0.0200
Autoformer-0.1764/0.05040.2703/0.12060.2309/0.09190.2931/0.1353
Nonstat.-0.1300/0.02960.1907/0.06590.0726/0.01020.0886/0.0176
ETSformer-0.1497/0.03650.2440/0.09660.1393/0.03350.2788/0.1346
LightTS-0.1307/0.03110.1900/0.06620.0744/0.01070.0928/0.0205
Informer-0.2312/0.08340.2750/0.12410.1058/0.01890.1201/0.0273
Reformer-0.2387/0.09160.2822/0.12980.1516/0.04080.2112/0.0812
LLM (PT, no VQ)L0 (-)0.1593/0.04520.2925/0.16220.1134/0.02590.3141/0.2204
VQ-LLMTSL0 (-)0.1324/0.03230.2043/0.08420.1109/0.01750.4310/0.4374
VQ-LLMTS-RegL0 (-)0.1354/0.03240.1873/0.06310.0755/0.01130.1014/0.0230
VQ-LLMTS-Reg + PatchTST (2-stage)Pred L20.1330/0.03140.1954/0.07020.0768/0.01160.1014/0.0231

Part 1b: Elec/ILI/Traffic/Weather (MAE/MSE)

MethodFuture textElectricityILITrafficWeather
Time-LLMOracle L20.2341/0.09822.2189/8.52500.2724/0.17300.0159/0.0005
GPT4TS-0.2062/0.08111.1460/3.44380.2199/0.13300.0182/0.0007
S2IP-LLM-0.2055/0.07861.3009/3.83300.2204/0.13860.0185/0.0007
Chronos-0.2197/0.09771.4715/5.20680.1328/0.10080.0127/0.0004
Timer-1.0026/1.24262.2685/8.51780.8237/0.98300.0235/0.0010
DLinear-0.2114/0.08641.5081/4.59150.3166/0.23000.0255/0.0013
PatchTST-0.2373/0.09501.1683/3.38530.2119/0.13100.0165/0.0006
TimesNet-0.2013/0.07621.1708/3.61220.1683/0.11580.0188/0.0007
FEDformer-0.2352/0.10071.3730/4.50060.2749/0.18640.0341/0.0020
Autoformer-0.2313/0.09791.9311/7.30230.2139/0.14520.4149/0.2341
Nonstat.-0.2641/0.12611.4807/5.43650.2319/0.16350.0220/0.0009
ETSformer-0.2808/0.14001.5645/5.31890.4467/0.37180.0757/0.0094
LightTS-0.2124/0.08801.5199/4.61480.2862/0.20070.0252/0.0013
Informer-0.2564/0.11311.7666/8.03900.2406/0.16750.0335/0.0019
Reformer-0.3412/0.19851.7121/7.96500.3541/0.30190.0388/0.0028
LLM (PT, no VQ)L0 (-)0.3204/0.29311.6674/5.99060.3222/0.43470.0189/0.0009
VQ-LLMTSL0 (-)0.2318/0.10721.3391/5.03000.1334/0.13810.0162/0.0006
VQ-LLMTS-RegL0 (-)0.2094/0.07922.0127/7.39870.1665/0.10740.0136/0.0005
VQ-LLMTS-Reg + PatchTST (2-stage)Pred L20.2120/0.08181.9587/7.31730.1694/0.10940.0137/0.0005

Part 2: M4 (OWA)

MethodFuture textYrQtrMoWkDayHrW.Avg
Time-LLMOracle L20.91761.13581.13371.99941.17241.85801.0922
GPT4TS-0.98751.09851.05581.71011.14333.30041.0657
S2IP-LLM-0.81700.90430.90301.71581.05261.11420.8937
Chronos-0.83120.88360.86591.11621.02780.49620.8683
Timer-2.00052.18422.12162.66591.96801.21202.1005
DLinear-0.87040.97570.95421.46901.11671.16630.9497
PatchTST-0.84720.98610.98771.49621.31323.11160.9794
TimesNet-0.90431.01521.02251.45471.20972.62771.0097
FEDformer-0.87261.06681.07851.73901.60483.32661.0623
Autoformer-0.88681.10361.42961.74431.73802.63591.2457
Nonstat.-0.90241.07451.08351.87761.16023.83231.0572
ETSformer-1.12371.01261.04611.17311.20061.50611.0648
LightTS-0.87230.98471.02151.54961.12331.46370.9864
Informer-0.88391.04551.15311.55521.23544.78951.0853
Reformer-0.97151.14381.16601.93361.29923.47471.1339
LLM (PT, no VQ)L0 (-)1.51250.87190.98771.15181.08400.68391.0840
VQ-LLMTSL0 (-)1.02021.38391.87911.28752.00340.65151.5608
VQ-LLMTS-RegL0 (-)0.87910.86310.83920.73011.18500.87690.8685
VQ-LLMTS-Reg + PatchTST (2-stage)Pred L20.96360.84740.76940.70231.13900.83910.8485

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.