- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper deals with the time-series forecasting problem with large vision models. Specifically, it focuses on the unexamined premise, i.e., rendered time-series images are close to natural images. And then it identifies two gaps: spectral and structural gaps.
Based on the above observations, this paper addresses the two gaps with a two-branch model to bridge the gaps.
A Spectral-Aware Adaptation Branch mitigates the spectral gap, while a Structural-Guided Adaptation Branch mitigates the structural gap. Finally, a fusion module is designed to combine the prediction of both branches.
Experiments are conducted on seven real-world benchmarks and compared with strong LVM- and LLM-based baselines for both full-shot and few-shot settings. On the majority of settings/configurations, the proposed method outperforms existing methods.
Critical strengths
This paper is well-written and well-structured, providing good insights for vision-based time-series forecasting.
- It raised the unexamined premise of adopting large vision models (LVMs) for time-series forecasting, i.e., the time-series rendered images are not guaranted to match the data distribution and characteristics of natural images (pre-training data of LVMs). Thus, without this premise, time-series rendered images may have a gap with natural images.
- From the above unexamined premise, this paper identifies two gaps: spectral and structural gaps. The first one is based on the power spectrum model, while the second one mentions the spurious spatial adjacencies and continous temporal adjacencies issues. Both of them are real problems of the LVMs-based time-series forecasting.
The two-branch module designs are consistent with the identified gaps, and aims for addessing the gaps.
- According to the power spectra analysis, this paper proposes Spectral-Aware Adaptation Branch to adjust the magnitude of input time-series images. This moves their power spectral closer to the training image spectral, which making them more suitable to be input to LVM.
- Structural-Guided Adaptation Branch has two designs: 1D temporal indices embedding, and LoRA finetuning. It somehow mitigates the structral gap of time-series images and natural images.
Experimentally, the propposed method outperforms existing VLM and LLM-based time-series forecaster.
- This is shown in Table 1 (Full) and Table 2 (Few-shot).
- Ablation study validates the effectiveness of the proposed modules (Table 3).
Critical weaknesses
While this paper identifies two important gaps of vision-based time-series forecasting, the module designs and their experiments are not always relevant and convincing w.r.t. the two gaps.
- First, Spectral-Aware Adaptation Branch is designed to amplify the magnitude of time-series images, thus pushing the power spectrum of rendered images to that of natural images. This is a reasonable design, but the experimental verification is less convincing. $\Delta \alpha \approx$$0.53$ for natural-rendered gap, whilt Table 4 only shows 0.1 gap improvement. This is not quite consistent with the assumption.
- Second, the second gap is the spurious relation and discontinouty of rendered time-series images. What Temporal Grounding Adapter introduces is 1D position embedding (PE) + projection + LoRA. It is not quite clear how replacing 2D PE with 1D mitigates above gap, and what are the effects of projection + LoRA.
Continuing above point, is there a direct 1D vs. 2D PE comparison? It seems Table 3 only albates the whole module.
The experiments have missed some comparison method, and inference efficiency comparison/analysis.
- DMMV is the second best model in Table 1. Why it is not compared in Table 2?
- In Figure 8, the proposed SSDA method has best MSE, while the number of parameters lie in the middle and memory consumption is the largest. How about the real inference time compared with other baselines?
Questions for authors
My questions are mainly about the two gaps assumption and the relationship with designed modules/experimental verifications. And in addition, the inference time and missing baseline comparison for few-shot setting should be considered.
4 comments
2. The temporal indexing of TGA, the roles of projection and LoRA, and the 1D vs. 2D PE comparison
What Temporal Grounding Adapter introduces is 1D position embedding (PE) + projection + LoRA. It is not quite clear how replacing 2D PE with 1D mitigates above gap,
Each patch corresponds to a group of samples rather than a single sample. Before resizing, each column contains one complete, contiguous period, and each row marks a position within that period. Resizing merely stretches this correspondence proportionally onto the pixel grid. A patch therefore covers a rectangular block of "phase interval by period interval", and near the boundary it may span several columns. For example, a patch near the top of the image covers samples near the beginning of each period it spans.
MAE reads patches in row-major order by default (left to right, then down), which conflicts with a layout where time advances down the columns. TGA instead reads the patch grid from top to bottom, then moves to the next column on the right. Thus, patch index (counting from zero) refers to row and column : the column sets the periods, the row sets the phase, giving each patch an approximate location in the original series.
This is a coarse-grained temporal positional encoding that gives an approximate rather than exact sample index. What it does restore is the correct order of the series -- adjacent indices correspond to adjacent phases, and crossing columns corresponds to crossing periods, so the patch read-out order is strictly aligned with the original temporal order. The ablation study and the reconstruction visualization of Fig. 4 confirm that this recovery is effective.
and what are the effects of projection + LoRA.
The projection maps the sinusoidal temporal encoding into the patch-embedding space; LoRA low-rank fine-tunes the attention projection matrices. To observe their effects more directly, we additionally ablate each component of the structural branch (averaged over ):
| Method (↓), Metric (→) | ETTh1 | ETTh2 | ETTm1 | ETTm2 | Avg |
|---|---|---|---|---|---|
| (1) Full SSDA (dual-branch) | 0.387 0.406 | 0.327 0.376 | 0.332 0.363 | 0.249 0.311 | 0.324 0.364 |
| (2) Single-branch Structural branch (TGA + projection + LoRA) | 0.391 0.408 | 0.332 0.378 | 0.342 0.367 | 0.260 0.318 | 0.331 0.368 |
| Drop | −1.03% −0.49% | −1.53% −0.53% | −3.01% −1.10% | −4.42% −2.25% | −2.32% −1.03% |
| (3) Single-branch MAE + TGA (no LoRA, no projection) | 0.396 0.412 | 0.335 0.384 | 0.346 0.368 | 0.261 0.321 | 0.335 0.371 |
| Drop | −2.33% −1.48% | −2.45% −2.13% | −4.22% −1.38% | −4.82% −3.22% | −3.32% −1.99% |
The results show that removing the projection and LoRA (row 3) degrades the average MSE by only 3.32%, while the full structural branch that retains TGA (row 2) degrades it by 2.32%, and the two gaps are close. This indicates that TGA is the backbone of the structural branch: its 1D temporal index provides the irreplaceable temporal-order prior, on top of which the projection and LoRA perform fine-grained adjustment, the former mapping the prior into the patch-embedding space and the latter letting attention reorganise the patches accordingly.
Continuing above point, is there a direct 1D vs. 2D PE comparison? It seems Table 3 only albates the whole module.
As suggested, we add the direct 1D vs. 2D comparison. 1D is TGA's temporal index; 2D is the patch's spatial coordinate on the image grid (row-major). The 2D control is implemented by taking the patch row and column , applying the standard 1D sinusoidal encoding to each, and concatenating the row and column encodings along the last dimension into a positional tensor.
Results (each dataset averaged over ; single-branch, both without SMA, to isolate the PE itself; same hyper-parameters as Table 3):
| Method | ETTh1 | ETTh2 | ETTm1 | ETTm2 | Avg |
|---|---|---|---|---|---|
| MSE MAE | MSE MAE | MSE MAE | MSE MAE | MSE MAE | |
| (1) SSDA (full dual-branch) | 0.387 0.406 | 0.327 0.376 | 0.332 0.363 | 0.249 0.311 | 0.324 0.364 |
| (2) Single-branch Structural branch (1D TGA) | 0.391 0.408 | 0.332 0.378 | 0.342 0.367 | 0.260 0.318 | 0.331 0.368 |
| Drop | −1.03% −0.49% | −1.53% −0.53% | −3.01% −1.10% | −4.42% −2.25% | −2.32% −1.03% |
| (3) Single-branch MAE + 2D positional encoding | 0.394 0.409 | 0.341 0.382 | 0.353 0.386 | 0.265 0.323 | 0.338 0.375 |
| Drop | −1.81% −0.74% | −4.28% −1.60% | −6.33% −6.34% | −6.43% −3.86% | −4.48% −3.02% |
Replacing TGA's 1D temporal index by the 2D spatial PE costs a further 2.16% in average MSE (1.99% in MAE) on top of the structural branch, and the 1D index wins on all four datasets. This confirms that TGA's 1D temporal positional encoding is what restores the 1D temporal features of the series.
3. On the missing DMMV in Table 2 and inference time
We thank the reviewer for raising these questions. Detailed responses are given below.
DMMV is the second best model in Table 1. Why it is not compared in Table 2?
On the few-shot comparison, DMMV's original paper reports no few-shot results, and DMMV performed poorly in few-shot in our reproduction, so we did not include it as a baseline in the few-shot experiments. Taking ETTh1 as an example, we additionally ran DMMV under the 10% training-data protocol, and the results are as follows:
| Dataset | H | SSDA MSE | SSDA MAE | DMMV MSE | DMMV MAE |
|---|---|---|---|---|---|
| ETTh1 | 96 | 0.359 | 0.376 | 0.6706 | 0.5744 |
| ETTh1 | 192 | 0.389 | 0.403 | 0.7231 | 0.6017 |
| ETTh1 | 336 | 0.417 | 0.420 | 0.7692 | 0.6326 |
| ETTh1 | 720 | 0.418 | 0.442 | 0.7931 | 0.6542 |
With 10% data, SSDA's MSE stays at 0.359-0.418 while DMMV reaches 0.6706-0.7931, a relative gap of about 46%, with DMMV's MSE far above even its own full-shot level (~0.34). This shows that DMMV needs abundant data and training to reach its best performance, whereas SSDA remains robust in the low-data regime.
The number of parameters lie in the middle and memory consumption is the largest. How about the real inference time?
As requested, we add inference time for each model. We measure the inference time under identical conditions (ETTh1, H=96, batch=32), and the results are as follows:
| Model | 1st count | Memory | Parameter | Inference time (s/epoch) |
|---|---|---|---|---|
| TimeLLM | 1 | 64.88G | 76652.92K | 118.6 |
| VisionTS | 6 | 52.07G | 54.50K | 18.5 |
| DMMV | 16 | 129.22G | 189.60K | 41.5 |
| SSDA | 48 | 104.91G | 1496.55K | 37.2 |
As the table shows, relative to the second-best model DMMV, SSDA uses both less memory (104.91G vs. 129.22G) and less inference time (37.2 vs. 41.5) while taking 48 first places against DMMV's 16. SSDA does consume more computational resources than the single-branch VisionTS, but given 48 first places against VisionTS's 6, this extra cost is entirely acceptable.
Thank the authors for the detailed response. Now my concerns has been addressed.
I keep my score as Weak Accept.
1. Clarifying the role of the spectral-gap analysis and of SMA
We thank the reviewer for this comment and for recognising our design. Detailed responses are given below.
On the measurement of PSS, we compare the PSS of the rendered time-series images before and after the SMA module, and find that it increases on every dataset (ETTh1 1.37 → 1.48, ETTh2 1.98 → 2.06, ETTm1 1.68 → 1.78, ETTm2 1.62 → 1.68), i.e. moving towards natural images. One point needs clarification: PSS is not monotonically related to forecasting accuracy, as analysed in detail in App. A.5. PSS serves as a channel for observing and analysing a feature. A higher α means energy decays faster with frequency, i.e. less high-frequency (noise-like) content and more low-frequency (trend-like) content, while a lower α means the opposite. The observed increase therefore shows that SMA strengthens the trend features of the series, and the trend is precisely what a pre-trained MAE forecasts poorly. Conversely, pushing the PSS of the rendered images blindly towards the natural-image value may instead damage the series' own characteristics.
On the empirical results, the ablation study shows that the two branches work best in combination: over the 96/192/336/720 horizons on ETT, keeping only the spectral branch (SMA) raises MSE by 2.50%, while keeping only the structural branch (SG-LoRA + TGA) raises it by 7.60%, relative to the full SSDA. The visualisation (Fig. 4) corroborates this: the spectral branch yields smoother reconstructions without pronounced line structure, consistent with a trend-dominated signal, whereas the structural branch yields vertical patterns, consistent with 1D temporal ordering. Together they establish the complementarity of the two branches and show that SMA does strengthen the trend features of the rendered images.
In summary, SMA is not trained with an explicit natural-image spectral-matching loss; it is trained end-to-end for forecasting, so as to learn the latent frequency-domain characteristics of the series. PSS is not our learning objective but a way of observing the temporal features. The shift of PSS towards natural images results from SMA strengthening the trend component, which benefits forecasting.