- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents SAGE (Seeing and Augmenting with Grounded Encoding), a CLIP-based framework for time-series forecasting. It designs variable-specific augmentation mechamism to inject knowledge to CLIP models for time-series forecasting. Specifically, it proposes template-based knowledge injection with variable-wise gating, frequency-enhanced encoding and cross-variate context, and training-only visual alignment modules.
Experiments are conducted on eight long-term benchmarks, M4, and component analysis. Compared with non-LLM or non-VLM based methods, the proposed method achieves better or competitive performance.
Critical strengths
While most current works focus on LLMs, large vision models, or large vision-language models, this paper only leverages the relatively light-weight CLIP text encoder for time-series forecasting.
- It is free-from LLM calling, thus being potentially more efficient and lightweight (no experimental evidence at inference).
- Vision encoder is only used during training for contrastive learning and not used during forecasting, making the model lightweight.
- Although without LLM calling, the proposed method is quite performing compared with other non-LLM or non-VLM methods as shown in the experiments.
It designs useful knowledge injection mechasim to augment the time-series for efficient forecasting.
- Semantic, behaviour, relation, and temporal aspects are designed and incorporated to convert time-series into CLIP text encoder.
- Albation study verifies the effectiveness ot the text augmentation (2.8%)
Critical weaknesses
Although no LLM calling is an advantage of the proposed method, it should discuss or compare with LLM/VLM/LVM methods, at least as reference or upper-bound.
- Related works such as Time-LLM, OccamVTS, TimesCLIP, are relevant methods and SOTA methods in the benchmarks. It is better to compare with them and disucss the results/efficiency. Otherwise, it is unclear what is the benefits of no LLM calling.
- No LLM calling potentially has both better training and inference speed. However, this paper only mentioned the training time of each epoch. There is no comparison with exisiting LLM/non-LLM methods for training, and more importantly, no comparison for the inference efficiency w.r.t. time, speed, memory, parameter, etc. It is unclear whether no LLM calling introduces extra benefits for inference speed.
- In the ablation study, how about the performance of no text + no vision? This is the pure capability of text encoder.
Knowledge-injection design is the main contribution of the proposed method. But existing method such as Time-CLIP or Time-LLM and (Jin et al., 2024; Pan et al., 2024) already explored this direction.
- This paper should discuss the connection and differences to the existing knowledge-injection methods and compare with them.
Questions for authors
How about the inference time and memory consumption of the proposed method compared with both non-LLM and LLM-based method?
Can you compare with other knowledge-injection method?
Others, please refer to above weaknesses.
3 comments
Thank the authors for the rebuttal.
The rebuttal adds new experiments such as LLM/VLM comparison, and inference speed.
The comparison with LLM/VLM does not show the advantage of the proposed method. The inference speed comparison with TimeCMA, Time-VLM and Time-LLM suggest better inference time. However, no efficiency comparison with non-LLM method.
Overall, the rebuttal only partially responds to my concerns, and most concerns remain unresolved. So at this stage, I tend to keep my original score.
We apologize that our previous response did not fully address your comment: it did not include the inference cost of the non-LLM baselines. These measurements were available from our earlier experiments, and we report them below. All models were measured on one NVIDIA A800 GPU with a batch of 32 windows and L = H = 96.
1. Inference cost of the non-LLM baselines (latency in ms / peak memory in MiB)
| Model | ETTh1 (N=7) | Weather (N=21) | Traffic (N=862) |
|---|---|---|---|
| DLinear | 0.3 / 17 | 0.2 / 19 | 0.3 / 110 |
| TimesNet | 15.2 / 33 | 6.8 / 38 | 241 / 1,562 |
| PatchTST | 1.7 / 110 | 5.3 / 265 | 191 / 4,863 |
| iTransformer | 2.1 / 20 | 2.8 / 49 | 50 / 2,562 |
| SAGE | 28.9 / 376 | 66.8 / 586 | 215 / 2,107 |
Most of SAGE's cost comes from the shared 12-layer CLIP text Transformer, which accounts for 85% of the inference time and 89% of the FLOPs on average.
2. Balancing accuracy and cost
Taken together, these results show that SAGE strikes a favorable balance between accuracy and cost. Non-LLM models are cheaper, but SAGE is more accurate, with the lowest MSE on 7 of 8 datasets, while its cost stays at a few milliseconds per window and is comparable to PatchTST and TimesNet on high-dimensional data. LLM-based methods run a language model on every new window, which makes inference more than two orders of magnitude slower than SAGE, yet they bring no consistent accuracy gain.
We will add these results to the revision and state the efficiency claim in terms of this balance. We sincerely thank the reviewer for raising this point.
We sincerely thank Reviewer baTW for the positive assessment of SAGE's lightweight, LLM-free design and its knowledge-injection mechanism, and for the concrete questions. We answer each below.
W1. Comparison with LLM/VLM methods
We acknowledge the importance of including LLM/VLM-based baselines. Therefore, we conducted additional experiments with representative strong baselines such as Time-LLM, TimesCLIP, and Time-VLM. Please refer to our response to Reviewer 4xnE(W1) for the detailed results, which will also be updated in the revised manuscript.
W2. Inference and training efficiency against LLM and non-LLM methods
We agree that the efficiency claim needs measurements, not just training time per epoch. We measured every method with its official code on one A800, with batch 32 and L = H = 96. At inference we report the latency and peak memory.
Results. (ms per batch / peak MiB)
W3. Relation to and comparison with other knowledge-injection methods
We agree that knowledge injection itself is not new. SAGE’s distinction lies in what knowledge is injected, how it is fused, and how it is reused:
Finally, we thank Reviewer baTW for the valuable comments, and we will incorporate the newly reported experiments and data into the revised version of our manuscript.