- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This work investigates the lossless compression of numerical time series data based on large pre-trained foundational models. It aims to leverage the strength of such large pre-trained foundational models in realizing the zero-shot transfer of such compression capacity.
The main contribution is the framework which has bi-lateral branches for LLM probability coding and TSFM residual coding. The effectiveness of this framework is justified via integer and floating-point time series without dataset specific fine-tuning. The key idea is applying the next-token prediction into helping the time series forecasting. The validation of this presented framework is done based on 6 time series datasets.
Critical strengths
To me, I think the clear problem description and the motivation of studying this problem is one of the main strength. The bi-lateral framework for leveraging the large pre-trained foundational model is also clearly described.
Critical weaknesses
The most critical weakness is that the key idea connecting LLMs to data compression is no longer new. The previous study, e.g., Language Modeling is compression, the LLMZip etc., already present similar ideas. The difference in terms of applying tokenization into time series numerical data is thus incremental compared with existing ones.
Another critical weakness is that the theoretical part is a largely restatement of those already presented in previous studies.
The methodological development is majorly a simple bringing of existing concepts to a specific targeted problem without sufficient original development done by authors.
The validation is also insufficient due to the dataset coverage and baseline coverage.
Questions for authors
Q1. In Table 1, the time series foundational model based compressor seems quite ineffective, with compression ratios slightly higher than 1 in many cases. Does it indicate that the time series foundational model based compressor will undermine the framework presented?
Q2. The numerical results seem not consistently support the application effectiveness claim. It does not show sufficient convinces that the presented "outperform on most datasets"? Meanwhile, the dataset coverage is also limited.
Q3. Why related works is put after the numerical studies? This content organization is so weird.
2 comments
Response to Authors' Response
Thank you very much for your efforts in responding my pretty harsh comments. Indeed, time-series partnering with LLMs is flooding with too many staying at the level of simply taking other concepts and built it with another. Meanwhile, in the era of agentic research, the empirical results are no longer strong enough and we need to see the theory and science behind it. Sorry that I talked too much and hope you don't feel that my previous comments are too offensive.
Honestly, I do have some expectations in terms of technical in-depth and hope my feedback can help you move towards a higher quality work in the future.
W
We agree that the link between LLMs and data compression is established in LLMZip [m1] and Language Modeling Is Compression [m2]. Section 2.2 uses the known relation between negative log-likelihood and code length as background. LLMTime [m3] likewise studies numeric tokenization for zero-shot forecasting.
Our empirical question is how well two types of predictions from frozen models translate into compression gains on the same numerical time series without dataset-specific fine-tuning. The LLM path uses next-token probabilities directly in arithmetic coding. The TSFM path uses point forecasts and applies static arithmetic coding to their residuals.
We also compare numeric tokenizations while holding the GPT-2 checkpoint and series fixed. In Table 1, Text-Only gives a higher ratio than Number in all 18 pairwise comparisons across three GPT-2 sizes and six series. For example, on TH-GPS with GPT-2-large, the ratios are 4.013 for Number, 8.875 for Text-Only, and 9.236 for Space-Number.
Q1
We agree that several TSFM compression ratios are only slightly higher than 1. For example, Table 1 gives a best TSFM ratio of 1.111 on ZH-Root. The best TSFM ratio is below LZMA on five of six series. On TH-Climate, the best TSFM achieves 6.052, compared with 4.714 for value-level static arithmetic coding and 10.614 for LZMA. On USGS-Earthquakes, the best TSFM reaches 4.161, compared with 3.924 for LZMA and 4.753 for value-level static arithmetic coding.
We will present the TSFM branch as a comparison within the framework and distinguish its results from the compression gains observed with LLM probability coding. One possible explanation for the ratios near 1 is that small forecast errors can still produce many distinct residual symbols. If their frequencies are spread across those symbols, a static arithmetic coder has fewer opportunities to assign short codes. A follow-up could compare the TSFM against a simple delta predictor using the same residual representation and coder to assess how much the forecasts contribute.
Q2
We will qualify “outperform on most datasets” by specifying both the dataset and baseline coverage. Under shared-checkpoint accounting, the best LLaMA 2 configuration has the highest reported ratio among the methods in Table 1 on five of six series (City-Temp, TH-GPS, ZH-Root, TH-Climate, and Cyber-Vehicle). On City-Temp, one of the two public datasets, the best LLaMA 2 ratio is 7.081, compared with 4.312 for LZMA. On USGS-Earthquakes, the other public dataset, value-level static arithmetic coding achieves 4.753, compared with 4.682 for the best LLaMA 2 entry. This series has a small discrete domain and skewed value frequencies, which can favor static value coding. With four private series in the current evaluation, broader comparisons on public data against predictive codecs would help establish how often the gains carry over.
Q3
Thanks for the suggestion. We will move Related Work before the numerical studies and discuss LLM-based compression, numeric forecasting, and predictive residual coding there. This will make clear which ideas the framework draws on and what the comparisons in Table 1 are intended to assess.
References
[m1] Valmeekam et al. LLMZip: Lossless Text Compression using Large Language Models. arXiv:2306.04050, 2023.
[m2] Delétang et al. Language Modeling Is Compression. ICLR, 2024.
[m3] Gruver et al. Large Language Models Are Zero-Shot Time Series Forecasters. NeurIPS, 2023.