- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes VQ-LLMTS, a framework that integrates time series with LLMs by converting continuous time series into discrete tokens using RVQ, enabling time-series and textual tokens to be represented and generated within a shared autoregressive framework. The method consists of two stages: training an RVQ-based autoencoder to learn a discrete temporal vocabulary, followed by aligning the resulting time-series tokens with an LLM through a temporal projection module and parameter-efficient fine-tuning. The authors evaluate the framework on time-series understanding, short-horizon forecasting, the M4 benchmark, and context-aware forecasting, reporting competitive or improved results against several baselines.
Critical strengths
-
Time series are represented as discrete tokens rather than directly using continuous embeddings.
-
Residual Vector Quantization (RVQ) is employed to learn hierarchical representations of time-series patterns.
-
The training framework consists of two stages. The first stage aims to learn a discrete temporal vocabulary that captures diverse temporal patterns, while the second stage performs modality alignment between time-series representations and natural language.
Critical weaknesses
W1. In the main forecasting experiments, VQ-LLMTS, LLM-SFT, VQ-LLMTS-Reg, and Time-LLM use textual descriptions derived from the future target windows, while the other baselines do not receive this information. These descriptions contain information about future trends, fluctuations, and extrema locations, which is generally unavailable from historical observations alone. Therefore, the performance differences across these groups cannot be attributed solely to the RVQ representation. Although the appendix includes experiments without future-derived context, they are mainly limited to ETTh1, which is insufficient to establish whether the reported advantages hold across other datasets.
W2. The main contribution of this paper is to convert continuous time series into discrete tokens using RVQ and integrate them into the LLM vocabulary. However, the comparison between LLM-SFT and VQ-LLMTS changes both the numerical encoding and the representation interface, making it difficult to isolate the contribution of the discrete representation itself. Moreover, VQ-LLMTS-Reg jointly updates the regression head, LoRA weights, and temporal projector, without clearly separating their individual contributions to the performance improvement.
W3. The paper compares against several established forecasting models, but many of these baselines are relatively old. Comparisons with more recent and competitive time-series forecasting models and LLM-based approaches would help provide stronger evidence of the proposed method's competitiveness.
Questions for authors
Q1. I have concerns about the availability of future-derived textual prompts in practical forecasting and the fairness of the corresponding experimental comparisons. In the main forecasting experiments, the model receives not only historical time series but also textual descriptions derived from the ground-truth future window, including its trend, patterns of change, fluctuations, and locations of extrema, and uses this information at test time. However, in conventional forecasting settings where the future is unknown, these descriptions cannot be obtained in advance using the generation procedure described in the paper. Therefore, these experiments evaluate conditional forecasting with future information. When comparing against baselines that do not receive equivalent prompts, it is difficult to attribute performance differences entirely to the model design. Although the authors disclose this setup and provide L0 results without future-derived prompts on ETTh1 in the appendix, this supplementary analysis on a single dataset remains insufficient to comprehensively assess the method’s performance under conventional forecasting conditions, while the main forecasting results and related discussion in the paper remain based on settings in which target-derived prompts are provided.
Q2. Could the authors provide additional experimental comparisons with recent related methods to more comprehensively assess the effectiveness of the proposed method and its performance relative to existing approaches?
Q3. Regarding the design of the RVQ module, could the authors further clarify its contributions beyond existing VQ/RVQ methods? Specifically, what substantive improvements does this work introduce to the encoder, residual quantization mechanism, or training objective to address the characteristics of time series? Compared with methods such as TOTEM, which use VQ-VAE to learn discrete time-series representations, what advantages do these design choices offer, and what experimental results support these advantages?
4 comments
Response to Authors
Thank you for the detailed response and additional experiments.
-
Could the authors confirm whether a revised PDF has been uploaded? I find it difficult to identify the specific changes.
-
Table R3 presents extensive experimental results. However, the method described in the manuscript uses LoRA. To my understanding, comparable LLM-based time-series forecasting methods generally keep the LLM frozen. Therefore, I view LoRA as an additional adaptation component, and this aspect has not been explained or discussed. Furthermore, the comparison between Single-VQ and RVQ still does not quantify the incremental contribution of RVQ over Single-VQ to downstream forecasting under the same LLM framework.
Since the authors have addressed several questions together, I am unsure whether my observations are correct. In particular, please correct me if my understanding regarding LoRA is mistaken.
- Although the additional experiments strengthen the evidence for the method’s effectiveness, I believe that, as presented in the current PDF, the paper’s substantive methodological contribution beyond existing VQ-based time-series representation methods and unified multimodal frameworks remains insufficiently clarified. Based on the evidence currently provided, I am maintaining my original rating of 2 for now.
Thank you for your comments. We clarify the three points below.
- Revised PDF and changes
Yes, we have prepared a revised PDF and will upload it together with this response. To make the changes easier to identify, the revised manuscript:
- adds TOTEM, ChatTime, history-only L0, and PatchTST-predicted L2;
- explicitly describes the LoRA, projector, generation-head, and regression-head adaptation settings;
- presents the context definitions before the corresponding results;
- includes the reconstruction comparisons, full L0–L4 study, text-perturbation results, and retail case in clearly identified tables.
2.1 LoRA and adaptation
You are right that the original backbone weights are frozen while LoRA adapters are trained. Thus, LoRA provides parameter-efficient adaptation without updating all original backbone parameters.
Stage II also updates the temporal projector and expanded generation head. VQ-LLMTS-Reg further adapts LoRA and the temporal projector together with its regression head. These are deliberate choices for adapting the temporal vocabulary and forecasting interface, rather than leaving the entire model unchanged. We have now stated them explicitly in Section 4.3 and Appendix A.1.1.
LoRA itself is not our claimed novelty. Our contribution is the temporal-vocabulary representation and its integration with the LLM.
2.2 RVQ and its downstream benefit
Single-VQ and RVQ both use discrete temporal representations. RVQ additionally models the residual left by the first codebook, preserving more local variations and fine-grained temporal structure. Downstream learning therefore receives less lossy discrete evidence about the trajectory, which provides a direct mechanism for improved forecasting.
The reconstruction results support this mechanism:
- RVQ reduces reconstruction MAE from 0.5796 to 0.1344, a 76.8% reduction.
- Truncating the trained RVQ to its first codebook gives MAE 0.6632, much worse than the full two-stage result. This shows that the residual stage supplies useful additional information rather than being redundant.
Thus, RVQ is beneficial not merely because it reconstructs better, but because it preserves more task-relevant temporal information for discrete forecasting.
- Methodological contribution
Our contribution is the integration of learned hierarchical temporal codes into the LLM’s input and output vocabulary, with joint text–time-series alignment and a shared generation head for textual descriptions and temporal codes. A separately adapted regression variant provides the numerical forecasting interface.
This differs from:
- TOTEM, which does not integrate its VQ representations into an LLM text vocabulary;
- ChatTime, which uses fixed scalar discretization rather than learned residual temporal codes.
The revised introduction and related-work section now make these distinctions explicit.
Thank you again for helping us clarify the method and its contribution.
Response to Authors
Thank you for your response. Considering both the paper’s core contributions and experimental validation, I will maintain my current rating for now. I will participate in the upcoming discussion and reconsider my assessment in light of the other reviewers’ views.
Note: All shared supplementary tables (Tables R1–R5) are provided in our Global Response above.
W1 & Q1
Please see our response to the second reviewer (Reviewer LgVB, W2 & Q2) and Tables R3–R5. We added systematic experiments on practicality and fairness without ground-truth future text, including: (1) history-only L0 forecasting across all 14 datasets (8 standard + 6 M4 groups); (2) a two-stage pipeline predicting future windows via PatchTST from history alone to extract qualitative text; and (3) fixed-Oracle L4 text-noise ablations, Hourly/ILI non-monotonicity analyses, and real-world holiday forecasting with known operational priors.
W2 & Q2
We compared against Continuous VAE to directly test continuous versus discrete representations on downstream forecasting (Table R2): under the identical Qwen2.5 backbone, regression interface, and L2 text input, despite Continuous VAE's lower raw reconstruction error (MAE 0.04589 vs. 0.1344), VQ-LLMTS-Reg beats Continuous VAE + Qwen2.5 on 13/14 datasets (8/8 on Matrix A; M4 weighted OWA 0.7460 vs. 0.8189), strictly isolating the contribution of discrete RVQ vocabulary alignment.
We compared reconstruction fidelity and cost across discretization standards (Table R1): on 100k synthetic series, 2-layer RVQ reduces reconstruction MAE from 6.1252 (fixed-bin) and 0.5796 (Single-VQ) to 0.1344. Comparing generative and regression heads under L0 (Table R3), the regression head reduces MAE on smooth series (e.g., ETTm1 0.1109→0.0755, M4 Quarterly OWA 1.3839→0.8631), whereas the generative head excels on spiky/high-frequency series (e.g., ILI 2.0127→1.3391, M4 Hourly OWA 0.8769→0.6515).
W3 & Q3
We added comparisons with two recent baselines, i.e. TOTEM (purely unimodal time-series VQ-VAE) and ChatTime (fixed-bin LLM vocabulary expansion), across the same 14 datasets (Tables R2 & R3), finding:
Beyond the benefits of two-layer RVQ for signal reconstruction, which reduces MAE and MSE by 76.8% and 92.7%, respectively, relative to single-VQ, we unify temporal and textual losses via Next-Token Prediction (NTP) during training, substantially strengthening cross-modal alignment and time-series understanding (Table 1: LLM-Judge 77.14, Regex Num 0.8821 vs. Chat-TS-14B 52.09 / 0.2711). Atop this aligned backbone, simply switching the output to a regression head flexibly adapts the model to accurate point forecasting.
Compared with TOTEM, our unified, vocabulary-expanded LLM backbone supports both time-series language understanding and text-conditioned forecasting, while outperforming TOTEM on 7/8 Matrix A datasets under L0 history-only inputs and on all 8 datasets under L2 inputs. Compared with fixed-bin ChatTime, our method outperforms it in both waveform reconstruction (MAE 0.1344 vs. 6.1252) and downstream forecasting performance, outperforming ChatTime on all 8 Matrix A datasets and achieving a better M4 weighted OWA under L0 (0.8685 vs. 0.8782).