acceptodds
Under review as a conference paper at ICLR 2027

SAGE: Variate-Wise Semantic Augmentation for Vision-Language Time Series Forecasting

Abstract

Time series forecasting models operate on raw numerical sequences, lacking the semantic knowledge that domain experts implicitly leverage, such as the physical meaning of each variable, its statistical behaviour, and its temporal dynamics. Recent efforts to bridge this gap fall into two camps. Some rely on large language models at inference time, which is computationally expensive. Others apply uniform textual prompts at the dataset level, ignoring the heterogeneous semantics across individual variates. We propose SAGE (Seeing and Augmenting with Grounded Encoding), an end-to-end CLIP-based framework that jointly models temporal, cross-variable, textual and visual information. The CLIP text encoder processes frequency-enhanced patches and variable tokens, while gated residual paths inject variable-specific descriptions and statistical descriptors. In parallel, the frozen CLIP vision encoder aligns rendered series with temporal representations through a training-only contrastive objective. This dual use of CLIP adds complementary semantic and visual supervision without placing an LLM in the forecasting loop. Across eight long-term benchmarks and M4, SAGE achieves state-of-the-art accuracy. Ablations confirm complementary gains from multimodal alignment and variable-level knowledge.

Primary area:
learning on time series and dynamical systems
Keywords:
Deep Learning, Time Series Forecasting, Multi-modal

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_baTWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SAGE (Seeing and Augmenting with Grounded Encoding), a CLIP-based framework for time-series forecasting. It designs variable-specific augmentation mechamism to inject knowledge to CLIP models for time-series forecasting. Specifically, it proposes template-based knowledge injection with variable-wise gating, frequency-enhanced encoding and cross-variate context, and training-only visual alignment modules.

Experiments are conducted on eight long-term benchmarks, M4, and component analysis. Compared with non-LLM or non-VLM based methods, the proposed method achieves better or competitive performance.

Critical strengths

While most current works focus on LLMs, large vision models, or large vision-language models, this paper only leverages the relatively light-weight CLIP text encoder for time-series forecasting.

  • It is free-from LLM calling, thus being potentially more efficient and lightweight (no experimental evidence at inference).
  • Vision encoder is only used during training for contrastive learning and not used during forecasting, making the model lightweight.
  • Although without LLM calling, the proposed method is quite performing compared with other non-LLM or non-VLM methods as shown in the experiments.

It designs useful knowledge injection mechasim to augment the time-series for efficient forecasting.

  • Semantic, behaviour, relation, and temporal aspects are designed and incorporated to convert time-series into CLIP text encoder.
  • Albation study verifies the effectiveness ot the text augmentation (2.8%)
Critical weaknesses

Although no LLM calling is an advantage of the proposed method, it should discuss or compare with LLM/VLM/LVM methods, at least as reference or upper-bound.

  • Related works such as Time-LLM, OccamVTS, TimesCLIP, are relevant methods and SOTA methods in the benchmarks. It is better to compare with them and disucss the results/efficiency. Otherwise, it is unclear what is the benefits of no LLM calling.
  • No LLM calling potentially has both better training and inference speed. However, this paper only mentioned the training time of each epoch. There is no comparison with exisiting LLM/non-LLM methods for training, and more importantly, no comparison for the inference efficiency w.r.t. time, speed, memory, parameter, etc. It is unclear whether no LLM calling introduces extra benefits for inference speed.
  • In the ablation study, how about the performance of no text + no vision? This is the pure capability of text encoder.

Knowledge-injection design is the main contribution of the proposed method. But existing method such as Time-CLIP or Time-LLM and (Jin et al., 2024; Pan et al., 2024) already explored this direction.

  • This paper should discuss the connection and differences to the existing knowledge-injection methods and compare with them.
Questions for authors

How about the inference time and memory consumption of the proposed method compared with both non-LLM and LLM-based method?

Can you compare with other knowledge-injection method?

Others, please refer to above weaknesses.

3 comments

We sincerely thank Reviewer baTW for the positive assessment of SAGE's lightweight, LLM-free design and its knowledge-injection mechanism, and for the concrete questions. We answer each below.

W1. Comparison with LLM/VLM methods

We acknowledge the importance of including LLM/VLM-based baselines. Therefore, we conducted additional experiments with representative strong baselines such as Time-LLM, TimesCLIP, and Time-VLM. Please refer to our response to Reviewer 4xnE(W1) for the detailed results, which will also be updated in the revised manuscript.

W2. Inference and training efficiency against LLM and non-LLM methods

We agree that the efficiency claim needs measurements, not just training time per epoch. We measured every method with its official code on one A800, with batch 32 and L = H = 96. At inference we report the latency and peak memory.

Results. (ms per batch / peak MiB)

ModelETTh1 (N=7)Weather (N=21)Traffic (N=862)
TimeCMA3,638 / 75911,083 / 1,354430,044 (7.2 min) / 19,675
Time-VLM245 / 710188 / 711268 / 1,988
Time-LLM6,305 / 19,80519,193 / 19,788775,368 (12.9 min) / 19,817
SAGE28.9 / 37666.8 / 586215 / 2,107

W3. Relation to and comparison with other knowledge-injection methods

We agree that knowledge injection itself is not new. SAGE’s distinction lies in what knowledge is injected, how it is fused, and how it is reused:

  • Variable-specific knowledge. Time-LLM uses dataset/task prompts and window statistics. SAGE explicitly binds semantic, behavioral, and relational descriptors to individual variables, providing variable identity and longer-term context beyond the current window.
  • Reuse across windows. TimeCMA caches window-specific prompt representations. SAGE caches variable-level descriptors that are reused across unseen windows, eliminating repeated natural-language encoding while retaining input-dependent knowledge selection.
  • Structured injection. SAGE first selects among each variable’s knowledge views, then reads across variables’ descriptors. Per-variable residual gates control injection strength, while a separate numerical bypass supplies statistical features directly, reducing reliance on language embeddings to preserve quantitative information.

Finally, we thank Reviewer baTW for the valuable comments, and we will incorporate the newly reported experiments and data into the revised version of our manuscript.

Thank the authors for the rebuttal.

The rebuttal adds new experiments such as LLM/VLM comparison, and inference speed.
The comparison with LLM/VLM does not show the advantage of the proposed method. The inference speed comparison with TimeCMA, Time-VLM and Time-LLM suggest better inference time. However, no efficiency comparison with non-LLM method.

Overall, the rebuttal only partially responds to my concerns, and most concerns remain unresolved. So at this stage, I tend to keep my original score.

We apologize that our previous response did not fully address your comment: it did not include the inference cost of the non-LLM baselines. These measurements were available from our earlier experiments, and we report them below. All models were measured on one NVIDIA A800 GPU with a batch of 32 windows and L = H = 96.

1. Inference cost of the non-LLM baselines (latency in ms / peak memory in MiB)

ModelETTh1 (N=7)Weather (N=21)Traffic (N=862)
DLinear0.3 / 170.2 / 190.3 / 110
TimesNet15.2 / 336.8 / 38241 / 1,562
PatchTST1.7 / 1105.3 / 265191 / 4,863
iTransformer2.1 / 202.8 / 4950 / 2,562
SAGE28.9 / 37666.8 / 586215 / 2,107

Most of SAGE's cost comes from the shared 12-layer CLIP text Transformer, which accounts for 85% of the inference time and 89% of the FLOPs on average.

2. Balancing accuracy and cost

Taken together, these results show that SAGE strikes a favorable balance between accuracy and cost. Non-LLM models are cheaper, but SAGE is more accurate, with the lowest MSE on 7 of 8 datasets, while its cost stays at a few milliseconds per window and is comparable to PatchTST and TimesNet on high-dimensional data. LLM-based methods run a language model on every new window, which makes inference more than two orders of magnitude slower than SAGE, yet they bring no consistent accuracy gain.

We will add these results to the revision and state the efficiency claim in terms of this balance. We sincerely thank the reviewer for raising this point.

Reviewer_dN82OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

SAGE combines a CLIP text encoder with cached variable-specific descriptions, statistical features, and training-only visual alignment for time series forecasting. The paper evaluates the framework on eight long-term benchmarks and M4, reporting competitive accuracy and gains from descriptor augmentation and visual supervision.

Critical strengths
  1. The framework provides an explicit mechanism for incorporating variable meanings, training-set statistics, and cross-variable relationships through gated fusion. The separate statistical bypass offers a direct way to retain numerical descriptors.

  2. The proposed framework uses cached descriptions and training-only visual supervision, avoiding generative LLM calls and image encoding during forecasting.

  3. The evaluation covers several forecasting domains. The paper acknowledges limitations concerning seed variability, component attribution, and comparisons with closely related multimodal methods.

Critical weaknesses
  1. The incremental contribution over closely related methods is insufficiently evaluated

While SAGE introduces architectural differences from Time-LLM, its conceptual advance over existing language-assisted forecasting approaches, particularly the closely related TimesCLIP backbone, remains insufficiently established. Direct comparisons and targeted ablations would help clarify the contribution of the proposed variable-specific augmentation beyond the existing backbone.

  1. The roles of CLIP pretraining and numerical FFT features require clearer analysis

The paper explains how projected FFT features are incorporated into the fine-tuned CLIP text Transformer, but offers limited analysis of why this combination is effective. Ablating the numerical FFT branch and comparing fine-tuned CLIP with randomly initialized and frozen encoders would help distinguish the contributions of frequency features, pretraining, and fine-tuning.

  1. The contribution of semantic content requires further validation

The experiments compare descriptor families, but do not test whether meaningful descriptions outperform unrelated text, shuffled descriptions, or parameter-matched non-textual alternatives. Separate evaluations of the statistical bypass and text-alignment loss would also help clarify the source of the gains.

  1. The fixed input history limits the scope of the performance conclusions

The experiments use a 96-step lookback, whereas the original PatchTST paper reports strong results with longer histories, including its 512-step configuration. Although 512 is not universally optimal, restricting the comparison to 96 steps may underestimate this baseline. Longer-history experiments would clarify whether SAGE’s advantage and descriptor gains persist.

Questions for authors

Please refer to the Critical Weaknesses section above.

1 comment

We sincerely thank Reviewer 2 for the thoughtful review. We answer the four concerns in order.

W1. Incremental contribution over TimesCLIP

We list exactly what SAGE changes and test it with a matched comparison. Relative to TimesCLIP, SAGE adds:

  • Per-variable cached descriptors (variable semantics and training-split statistics), fused through view attention, per-variable gates and a statistical bypass. TimesCLIP has no information about what each variable is.
  • A gated FFT branch on the patch tokens.
  • Additive rather than replacing fusion of the cross-variate token.

We compared TimesCLIP, the SAGE backbone without descriptors, and full SAGE under the same protocol. You can see the results in our reply to Reviewer 4xnE (W1).

W2. Roles of the FFT features and CLIP pretraining

We agree that the paper explained how the FFT features enter the model but not why they help. We cover the FFT branch here. Pretraining and fine-tuning are tested in our reply to Reviewer 4nxE (W3), which compares a fine-tuned, a frozen and a randomly initialized text encoder. To evaluate the gain of FFT module, we ablated the branch by retraining SAGE without it.

Results.

ETTh1ETTh2ETTm1ETTm2WeatherExchange
SAGE0.4400.3730.3840.2830.2430.332
w/o FFT branch0.4430.3760.3850.2840.2430.333

Analysis SAGE's FFT branch combines three components:

  • The Hann window removes the seam. We multiply each patch by a Hann window before the FFT. It fades both ends to zero, like a fade-in and fade-out on an audio clip, so there is no seam when the patch wraps around.
  • Overlapping patches recover the faded edges. Patches overlap by half (stride 8), so the edge of one patch lies near the centre of its neighbour.
  • Cross-attention lets neighbouring patches share spectral evidence. The original tokenizer processes each patch in isolation. In the FFT branch, each patch attends to the spectra of all patches in the window and adds a gated summary to its own token.

W3. Whether semantic content matters

We agree this is the key question for a descriptor-based method. For each text mode, we compared the trainable parameters its fusion layers add (the descriptor embeddings themselves are frozen buffers and add none) with its average MSE reduction relative to No Text, separately for the four ETT datasets and the other four datasets.

Results.

ModeAdded trainable params (fusion layers)MSE reduction, ETT datasetsMSE reduction, other datasets
+Stat1.3M2.0%0.9%
+Dyn2.1M1.3%0.6%
+S+D2.4M1.9%2.6%
+Dyn+Freq2.2M0.5%0.0%
+Full2.4M0.5%1.8%

The results show that larger fusion layers do not bring larger gains. This rules out model size as the explanation. We will add these controls, a learnable-embedding control of the same shape, and separate removal of the bypass and L_text to the revision.

W4. Fixed 96-step lookback

Our main comparison follows the L = 96 protocol, but this can understate methods such as PatchTST that benefit from longer history. We trained SAGE and SAGE without descriptors at L = 512 and compared them with each method's published L = 512 results.

Results.

DatasetPatchTSTTime-VLMSAGE w/o desc.SAGESAGE (L=96)
ETTh10.4130.4050.4220.4070.440
ETTm10.3530.3470.3560.3470.384
Weather0.2260.2240.2270.2200.243
ECL0.1590.1720.1810.1620.165
Traffic0.3910.4190.4600.4070.430

The results show that SAGE benefits from longer history. Moving from L = 96 to L = 512 lowers SAGE's MSE by 5–12%. We will add these results and the L = 512 setting to the revision.

Finally, we thank Reviewer dN82 for the valuable comments and for understanding the delay in our response caused by experimental issues. We hope that the explanations above address your concerns, and we will incorporate the newly reported results into the revised manuscript.

Reviewer_4xnEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SAGE, a CLIP-based time-series forecasting framework centered on variable-specific descriptor augmentation. It combines cached descriptions derived from metadata and training-set statistics with gated textual fusion and a numerical statistical bypass. The full configuration additionally uses frequency-enhanced patch encoding and training-only alignment with frozen CLIP visual features. A lightweight configuration omits patching and visual alignment for high-dimensional datasets. The reported results are competitive across eight long-term forecasting benchmarks and M4. However, the current experiments do not sufficiently establish whether the improvements arise from meaningful semantic augmentation, numerical feature engineering, pretrained representations, or additional model capacity. Direct comparisons with the closest related approaches are also missing.

Critical strengths
  1. Incorporating variable-specific metadata and statistical descriptors is well motivated, and caching descriptions avoids generative-model calls during forecasting.
  2. The manuscript states that cached statistics, correlations, and data-derived descriptions are constructed exclusively from the training partition.
  3. The paper includes long-term and short-term forecasting results, along with text-augmentation and visual-alignment ablations.
Critical weaknesses
  1. The manuscript explicitly identifies TimesCLIP as a closely related approach that already adapts CLIP’s text encoder to numerical patch and variate tokens, incorporates visual alignment, and uses cross-attention for variate representations. However, TimesCLIP is absent from the experimental comparison, as are TimeCMA and Time-VLM. Since SAGE’s central contribution is gated variable-specific augmentation within this existing framework, a matched comparison against the underlying backbone without the proposed augmentation is essential. Comparisons against numerical forecasters alone cannot establish the incremental value over closely related multimodal approaches.
  2. The descriptor pipeline combines variable meanings, verbalized statistics, a direct statistical bypass, gated fusion, and text-alignment supervision. Table 5 does not clearly identify which of these components are removed in the “No Text” condition. Consequently, the reported improvements cannot be attributed specifically to textual semantics. They may instead result from providing training-set statistics, additional variable identifiers, or additional trainable capacity.
  3. The visual ablation demonstrates that adding the alignment objective improves the reported runs, but it does not establish that pretrained visual representations are necessary. A randomly initialized frozen image encoder or numerical contrastive objective could help determine whether the gain reflects CLIP-specific knowledge or generic regularization. Similarly, the text Transformer is fine-tuned on numerical tokens after replacing its original positional embeddings and bypassing its word embeddings. A randomly initialized or non-pretrained Transformer with the same architecture would clarify the contribution of language pretraining.
Questions for authors

See Weaknesses.

1 comment

We sincerely thank Reviewer 4xnE for the careful and constructive review, and for recognizing the motivation for variable-specific descriptors and the benefit of caching them.We address each concern below.

W1. Missing comparison with TimesCLIP, TimeCMA, Time-VLM and the matched backbone

We added all three methods under our protocol, plus the matched backbone: SAGE with every descriptor component removed ("SAGE w/o desc.", identical to "No Text" in Table 5).

Results.

DatasetTimesCLIPTimeCMATime-VLMSAGE w/o desc.SAGE
ETTh10.4530.4230.4600.4500.440
ETTh20.3830.3720.3990.3810.373
ETTm10.3970.3800.3800.3950.384
ETTm20.2960.2750.2720.2940.283
ECL0.1680.1740.2220.1670.165
Traffic0.4370.4300.5150.4340.430
Weather0.2550.2500.2580.2530.243
Exchange0.3560.3550.3610.3540.332

W2. What "No Text" removes, and whether the gains are textual

"No Text" removes every descriptor component: the text cross-attention and its per-variable gates, the multi-view and time-slot views, the statistical bypass, and L_text. We agree that Table 5 was ambiguous,We now report every text mode, not only the best one.

DatasetNo Text+Stat+Dyn+S+D+Dyn+Freq+FullBest
ETTh10.4500.4410.4510.4430.4550.4550.440
ETTh20.3810.3790.3740.3750.3840.3820.373
ETTm10.3950.3840.3890.3860.3900.3910.384
ETTm20.2940.2860.2880.2870.2860.2870.283
ECL0.1670.1660.1650.1660.1660.1670.165
Traffic0.4340.4360.4330.4330.4360.4360.430
Weather0.2530.2440.2450.2450.2460.2470.243
Exchange0.3540.3540.3620.3320.3640.3350.332

W3. Whether pretrained visual and language representations are necessary

We retrained SAGE with each encoder replaced, changing nothing else: a frozen text encoder; a randomly initialized text encoder; a randomly initialized frozen vision encoder with the same InfoNCE; and both encoders randomly initialized (no pretraining).

Results.

VariantETTh1ETTm1Weather
SAGE0.4400.3840.243
Text encoder pretrained but frozen0.4620.4030.255
Text encoder randomly initialized0.4530.3960.250
Vision encoder randomly initialized0.4460.3890.244
Both encoders randomly initialized0.4590.4010.251

The results show that Language pretraining is worth about 3% in MSE, and randomly initializing both encoders costs 3–4.5%. We will revise Section 4.4 accordingly.

Finally, we sincerely apologize for the delayed response due to experimental issues. Thank you for your valuable comments and we will incorporate the above results into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.