acceptodds
Under review as a conference paper at ICLR 2027

SSDA: Bridging Spectral and Structural Gaps via Dual Adaptation for Vision-Based Time Series Forecasting

Abstract

Large vision models (LVMs) have recently proven to be surprisingly effective time series forecasters, simply by rendering temporal data as images. This success, however, rests on a largely unexamined premise: the rendered time series images are sufficiently close to natural images for knowledge in pre-trained models to transfer effectively. We argue that two gaps still remain, i.e., spectral and structural gaps, fundamentally limiting the potential of LVMs for time series forecasting. Spectrally, we systematically reveal that rendered time series images exhibit a markedly shallower power spectrum than the natural images LVMs are pre-trained to recognize. Structurally, reshaping 1D temporal sequences into 2D grids fabricates spurious spatial adjacencies while severing genuine temporal continuities, misleading the spatial inductive biases of pre-trained LVMs. To bridge these gaps, we propose SSDA, a dual-branch network that spectrally and structurally adapts to unlock the full potential of LVMs for time series forecasting. At the data level, a Spectral Magnitude Aligner (SMA) applies 2D FFT to selectively enhance the magnitude spectrum toward natural-image statistics while preserving phase. At the model level, a Structural-Guided Low-Rank Adaptation (SG-LoRA) injects position-aware temporal encodings into patch embeddings and adapts attention via low-rank updates. The two branches are further adaptively fused to produce the final forecast. Extensive experiments on seven real-world benchmarks demonstrate that SSDA consistently outperforms strong LVM- and LLM-based baselines under both full-shot and few-shot settings. Code is publicly available at https://anonymous.4open.science/r/SSDA-8C5B.

Primary area:
learning on time series and dynamical systems
Keywords:
Time Series Forecasting, LVM, MAE, LoRA, Adapters

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5bFoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper deals with the time-series forecasting problem with large vision models. Specifically, it focuses on the unexamined premise, i.e., rendered time-series images are close to natural images. And then it identifies two gaps: spectral and structural gaps.

Based on the above observations, this paper addresses the two gaps with a two-branch model to bridge the gaps.
A Spectral-Aware Adaptation Branch mitigates the spectral gap, while a Structural-Guided Adaptation Branch mitigates the structural gap. Finally, a fusion module is designed to combine the prediction of both branches.

Experiments are conducted on seven real-world benchmarks and compared with strong LVM- and LLM-based baselines for both full-shot and few-shot settings. On the majority of settings/configurations, the proposed method outperforms existing methods.

Critical strengths

This paper is well-written and well-structured, providing good insights for vision-based time-series forecasting.

  • It raised the unexamined premise of adopting large vision models (LVMs) for time-series forecasting, i.e., the time-series rendered images are not guaranted to match the data distribution and characteristics of natural images (pre-training data of LVMs). Thus, without this premise, time-series rendered images may have a gap with natural images.
  • From the above unexamined premise, this paper identifies two gaps: spectral and structural gaps. The first one is based on the power spectrum model, while the second one mentions the spurious spatial adjacencies and continous temporal adjacencies issues. Both of them are real problems of the LVMs-based time-series forecasting.

The two-branch module designs are consistent with the identified gaps, and aims for addessing the gaps.

  • According to the power spectra analysis, this paper proposes Spectral-Aware Adaptation Branch to adjust the magnitude of input time-series images. This moves their power spectral closer to the training image spectral, which making them more suitable to be input to LVM.
  • Structural-Guided Adaptation Branch has two designs: 1D temporal indices embedding, and LoRA finetuning. It somehow mitigates the structral gap of time-series images and natural images.

Experimentally, the propposed method outperforms existing VLM and LLM-based time-series forecaster.

  • This is shown in Table 1 (Full) and Table 2 (Few-shot).
  • Ablation study validates the effectiveness of the proposed modules (Table 3).
Critical weaknesses

While this paper identifies two important gaps of vision-based time-series forecasting, the module designs and their experiments are not always relevant and convincing w.r.t. the two gaps.

  • First, Spectral-Aware Adaptation Branch is designed to amplify the magnitude of time-series images, thus pushing the power spectrum of rendered images to that of natural images. This is a reasonable design, but the experimental verification is less convincing. $\Delta \alpha \approx$$0.53$ for natural-rendered gap, whilt Table 4 only shows 0.1 gap improvement. This is not quite consistent with the assumption.
  • Second, the second gap is the spurious relation and discontinouty of rendered time-series images. What Temporal Grounding Adapter introduces is 1D position embedding (PE) + projection + LoRA. It is not quite clear how replacing 2D PE with 1D mitigates above gap, and what are the effects of projection + LoRA.

Continuing above point, is there a direct 1D vs. 2D PE comparison? It seems Table 3 only albates the whole module.

The experiments have missed some comparison method, and inference efficiency comparison/analysis.

  • DMMV is the second best model in Table 1. Why it is not compared in Table 2?
  • In Figure 8, the proposed SSDA method has best MSE, while the number of parameters lie in the middle and memory consumption is the largest. How about the real inference time compared with other baselines?
Questions for authors

My questions are mainly about the two gaps assumption and the relationship with designed modules/experimental verifications. And in addition, the inference time and missing baseline comparison for few-shot setting should be considered.

4 comments
AuthorsOpenReview ↗

1. Clarifying the role of the spectral-gap analysis and of SMA

First, Spectral-Aware Adaptation Branch is designed to amplify the magnitude of time-series images, thus pushing the power spectrum of rendered images to that of natural images. This is a reasonable design, but the experimental verification is less convincing. δα=0.53\delta \alpha = 0.53 for natural-rendered gap, whilt Table 4 only shows 0.1 gap improvement. This is not quite consistent with the assumption.

We thank the reviewer for this comment and for recognising our design. Detailed responses are given below.

On the measurement of PSS, we compare the PSS of the rendered time-series images before and after the SMA module, and find that it increases on every dataset (ETTh1 1.37 → 1.48, ETTh2 1.98 → 2.06, ETTm1 1.68 → 1.78, ETTm2 1.62 → 1.68), i.e. moving towards natural images. One point needs clarification: PSS is not monotonically related to forecasting accuracy, as analysed in detail in App. A.5. PSS serves as a channel for observing and analysing a feature. A higher α\alpha means energy decays faster with frequency, i.e. less high-frequency (noise-like) content and more low-frequency (trend-like) content, while a lower α\alpha means the opposite. The observed increase therefore shows that SMA strengthens the trend features of the series, and the trend is precisely what a pre-trained MAE forecasts poorly. Conversely, pushing the PSS of the rendered images blindly towards the natural-image value may instead damage the series' own characteristics.

On the empirical results, the ablation study shows that the two branches work best in combination: over the 96/192/336/720 horizons on ETT, keeping only the spectral branch (SMA) raises MSE by 2.50%, while keeping only the structural branch (SG-LoRA + TGA) raises it by 7.60%, relative to the full SSDA. The visualisation (Fig. 4) corroborates this: the spectral branch yields smoother reconstructions without pronounced line structure, consistent with a trend-dominated signal, whereas the structural branch yields vertical patterns, consistent with 1D temporal ordering. Together they establish the complementarity of the two branches and show that SMA does strengthen the trend features of the rendered images.

In summary, SMA is not trained with an explicit natural-image spectral-matching loss; it is trained end-to-end for forecasting, so as to learn the latent frequency-domain characteristics of the series. PSS is not our learning objective but a way of observing the temporal features. The shift of PSS towards natural images results from SMA strengthening the trend component, which benefits forecasting.

AuthorsOpenReview ↗

2. The temporal indexing of TGA, the roles of projection and LoRA, and the 1D vs. 2D PE comparison

What Temporal Grounding Adapter introduces is 1D position embedding (PE) + projection + LoRA. It is not quite clear how replacing 2D PE with 1D mitigates above gap,

Each patch corresponds to a group of samples rather than a single sample. Before resizing, each column contains one complete, contiguous period, and each row marks a position within that period. Resizing merely stretches this correspondence proportionally onto the pixel grid. A patch therefore covers a rectangular block of "phase interval by period interval", and near the boundary it may span several columns. For example, a patch near the top of the image covers samples near the beginning of each period it spans.

MAE reads patches in row-major order by default (left to right, then down), which conflicts with a layout where time advances down the columns. TGA instead reads the 14×1414\times14 patch grid from top to bottom, then moves to the next column on the right. Thus, patch index ii (counting from zero) refers to row i mod 14i\bmod 14 and column ⌊i/14⌋\lfloor i/14\rfloor: the column sets the periods, the row sets the phase, giving each patch an approximate location in the original series.

This is a coarse-grained temporal positional encoding that gives an approximate rather than exact sample index. What it does restore is the correct order of the series -- adjacent indices correspond to adjacent phases, and crossing columns corresponds to crossing periods, so the patch read-out order is strictly aligned with the original temporal order. The ablation study and the reconstruction visualization of Fig. 4 confirm that this recovery is effective.

and what are the effects of projection + LoRA.

The projection maps the sinusoidal temporal encoding into the patch-embedding space; LoRA low-rank fine-tunes the attention projection matrices. To observe their effects more directly, we additionally ablate each component of the structural branch (averaged over H∈{96,192,336,720}H\in\{96,192,336,720\}):

Method (↓), Metric (→)ETTh1ETTh2ETTm1ETTm2Avg
(1) Full SSDA (dual-branch)0.387 0.4060.327 0.3760.332 0.3630.249 0.3110.324 0.364
(2) Single-branch Structural branch (TGA + projection + LoRA)0.391 0.4080.332 0.3780.342 0.3670.260 0.3180.331 0.368
Drop−1.03% −0.49%−1.53% −0.53%−3.01% −1.10%−4.42% −2.25%−2.32% −1.03%
(3) Single-branch MAE + TGA (no LoRA, no projection)0.396 0.4120.335 0.3840.346 0.3680.261 0.3210.335 0.371
Drop−2.33% −1.48%−2.45% −2.13%−4.22% −1.38%−4.82% −3.22%−3.32% −1.99%

The results show that removing the projection and LoRA (row 3) degrades the average MSE by only 3.32%, while the full structural branch that retains TGA (row 2) degrades it by 2.32%, and the two gaps are close. This indicates that TGA is the backbone of the structural branch: its 1D temporal index provides the irreplaceable temporal-order prior, on top of which the projection and LoRA perform fine-grained adjustment, the former mapping the prior into the patch-embedding space and the latter letting attention reorganise the patches accordingly.

Continuing above point, is there a direct 1D vs. 2D PE comparison? It seems Table 3 only albates the whole module.

As suggested, we add the direct 1D vs. 2D comparison. 1D is TGA's temporal index; 2D is the patch's spatial coordinate on the image grid (row-major). The 2D control is implemented by taking the patch row r=⌊j/14⌋r=\lfloor j/14\rfloor and column c=j mod 14c=j\bmod14, applying the standard 1D sinusoidal encoding PE1d(⋅)∈RD/2\mathrm{PE}_1d(\cdot)\in\mathbb{R}^{D/2} to each, and concatenating the row and column encodings along the last dimension into a positional tensor.

Results (each dataset averaged over H∈{96,192,336,720}H\in\{96,192,336,720\}; single-branch, both without SMA, to isolate the PE itself; same hyper-parameters as Table 3):

MethodETTh1ETTh2ETTm1ETTm2Avg
MSE MAEMSE MAEMSE MAEMSE MAEMSE MAE
(1) SSDA (full dual-branch)0.387 0.4060.327 0.3760.332 0.3630.249 0.3110.324 0.364
(2) Single-branch Structural branch (1D TGA)0.391 0.4080.332 0.3780.342 0.3670.260 0.3180.331 0.368
Drop−1.03% −0.49%−1.53% −0.53%−3.01% −1.10%−4.42% −2.25%−2.32% −1.03%
(3) Single-branch MAE + 2D positional encoding0.394 0.4090.341 0.3820.353 0.3860.265 0.3230.338 0.375
Drop−1.81% −0.74%−4.28% −1.60%−6.33% −6.34%−6.43% −3.86%−4.48% −3.02%

Replacing TGA's 1D temporal index by the 2D spatial PE costs a further 2.16% in average MSE (1.99% in MAE) on top of the structural branch, and the 1D index wins on all four datasets. This confirms that TGA's 1D temporal positional encoding is what restores the 1D temporal features of the series.

AuthorsOpenReview ↗

3. On the missing DMMV in Table 2 and inference time

We thank the reviewer for raising these questions. Detailed responses are given below.

DMMV is the second best model in Table 1. Why it is not compared in Table 2?

On the few-shot comparison, DMMV's original paper reports no few-shot results, and DMMV performed poorly in few-shot in our reproduction, so we did not include it as a baseline in the few-shot experiments. Taking ETTh1 as an example, we additionally ran DMMV under the 10% training-data protocol, and the results are as follows:

DatasetHSSDA MSESSDA MAEDMMV MSEDMMV MAE
ETTh1960.3590.3760.67060.5744
ETTh11920.3890.4030.72310.6017
ETTh13360.4170.4200.76920.6326
ETTh17200.4180.4420.79310.6542

With 10% data, SSDA's MSE stays at 0.359-0.418 while DMMV reaches 0.6706-0.7931, a relative gap of about 46%, with DMMV's MSE far above even its own full-shot level (~0.34). This shows that DMMV needs abundant data and training to reach its best performance, whereas SSDA remains robust in the low-data regime.

The number of parameters lie in the middle and memory consumption is the largest. How about the real inference time?

As requested, we add inference time for each model. We measure the inference time under identical conditions (ETTh1, H=96, batch=32), and the results are as follows:

Model1st countMemoryParameterInference time (s/epoch)
TimeLLM164.88G76652.92K118.6
VisionTS652.07G54.50K18.5
DMMV16129.22G189.60K41.5
SSDA48104.91G1496.55K37.2

As the table shows, relative to the second-best model DMMV, SSDA uses both less memory (104.91G vs. 129.22G) and less inference time (37.2 vs. 41.5) while taking 48 first places against DMMV's 16. SSDA does consume more computational resources than the single-branch VisionTS, but given 48 first places against VisionTS's 6, this extra cost is entirely acceptable.

Thank the authors for the detailed response. Now my concerns has been addressed.
I keep my score as Weak Accept.

Reviewer_wcCPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper looks at vision-based time series forecasting, where a series is folded by period into a 2D image and fed to a pre-trained MAE, as in VisionTS. The authors argue that these rendered images differ from natural images in two ways. The first is a spectral gap: the power-spectrum slope is lower (alpha about 1.66, against 2.19 for ImageNet). The second is a structural gap that comes from the periodic folding. To close them, SSDA uses two branches. The spectral branch passes the FFT magnitude through a small conv net, blends the output into the input with lambda = 0.05, and gives the result to a frozen MAE. The structural branch adds a gated sinusoidal patch-index encoding (TGA) and LoRA to the MAE. A clamped scalar beta mixes the forecasts of the two branches. The experiments cover 7 LTSF benchmarks in full-shot and few-shot settings, with ablations on the 4 ETT datasets, a PSS analysis before and after SMA, and a periodicity study.

Critical strengths
  1. The question of how period-folded time series images differ statistically from the natural images an MAE was trained on is relevant to the VisionTS line of work, and the PSS measurement gives a concrete, reproducible handle on it.
  2. The added components are small (a 2-layer conv on the FFT magnitude, one extra encoding, LoRA with r = 4), and the evaluation covers both full-shot and few-shot settings against recent LVM baselines (DMMV, Time-VLM, VisionTS).
  3. The paper openly reports that alpha does not correlate with forecasting accuracy, and code is provided.
Critical weaknesses
  1. The spectral-gap motivation is not connected to the method or to accuracy. The premise is that a pre-trained MAE works better when its input has natural-image spectral statistics, but this is never tested. SMA is trained only with the forecasting loss, nothing pushes alpha toward the natural-image value, and with lambda = 0.05 alpha moves only slightly (e.g., ETTh1 from 1.37 to 1.48). The paper's own analysis finds the best MSE at the smallest alpha. It is therefore unclear whether the "spectral gap" is a real cause of error or just a descriptive statistic. The text-vs-image comparison (alpha for ASCII-rendered text) also does not say anything about LLM forecasters, which do not consume rendered images.
  2. The structural branch does not address the structural gap it is designed for. The structural gap comes from periodic folding, which places time steps from different periods next to each other. TGA is said to map "each spatial patch back to its original 1D temporal index", but Eq. 9 uses the raster patch index T = [0, ..., L-1], i.e., the same order already encoded by the MAE positional embedding. After the column-wise folding and 224x224 resize, this index is not a time index. Conceptually, TGA is a second positional embedding plus LoRA, not a mechanism that restores temporal order.
  3. The source of the gains cannot be identified. The labels of the two ablation variants are swapped between Table 3 and Table 11, so it cannot be read which branch matters. Under the Table 11 labels, the spectral-only model is worse than plain VisionTS on ETTm1 and ETTm2. In addition, SSDA mixes a frozen-MAE forecast with a LoRA-tuned-MAE forecast, which is a two-model ensemble; the natural control (frozen VisionTS + LoRA-VisionTS mixed with beta, without SMA and TGA) is missing. Some averages in Table 1 also do not match their own per-horizon values (they match Table 11 instead), which suggests the tables were not regenerated from one run.
  4. The comparison protocol is not controlled and the claims exceed the evidence. SSDA uses look-back windows of 1440-2360, while the paper does not state the look-back or the source (rerun vs. copied) of the baseline numbers. No seeds or variance are reported, hyperparameters appear to be chosen on test results, and many margins over DMMV are 0.001-0.01, with DMMV better on several cells and SSDA losing in few-shot on Electricity and Traffic. Statements such as "consistent across all datasets" are not supported.
Questions for authors
  1. What is the argument that a frozen MAE should benefit from inputs whose power spectrum matches natural images, given that Table 6 shows the opposite trend? Would adding an explicit spectral-matching objective change accuracy?
  2. How does TGA recover temporal order after the column-wise folding and resize, and how does it differ in function from the existing MAE positional embedding?
  3. Which ablation labels are correct (Table 3 or Table 11)? How much of the gain remains against a beta-mixed ensemble of frozen and LoRA-tuned VisionTS without SMA and TGA?
  4. Is the spectral/structural gap framing specific to MAE, or does it also hold for other vision backbones (e.g., DINO, CLIP) used in this line of work?
  5. Which baseline numbers were rerun and at which look-back? Were hyperparameters selected on a validation split, and can you report results over several seeds?
6 comments
AuthorsOpenReview ↗

1. Clarifying the role of the spectral-gap analysis and of SMA

We thank the reviewer for raising these questions. Detailed responses are given below.

The spectral-gap motivation is not connected to the method or to accuracy....SMA is trained only with the forecasting loss, nothing pushes alpha toward the natural-image value

Our spectral-gap module is SMA: a learnable CNN on Fourier magnitude with phase preserved, learning frequency-domain features of the series. SMA is not trained with an explicit natural-image spectrum-matching loss but end-to-end for forecasting, yet it reduces the spectral gap.

Methodologically, once time series and images are converted to the frequency domain, they are composed of magnitude and phase. A change in the magnitude changes the texture / frequency statistics; while a change in the phase would affect the spatial structure of the data and cause its own characteristics to be largely lost. We therefore can only operate on the magnitude to shrink the frequency-domain difference between the two modalities, which is exactly what SMA does.

Empirically, PSS(α\alpha) increases on every dataset with SMA (ETTh1 1.37 → 1.48, ETTh2 1.98 → 2.06, ETTm1 1.68 → 1.78, ETTm2 1.62 → 1.68). A higher PSS means less high-frequency content, thus weakening the periodic / noise-like part and strengthening the trend. Since the trend is exactly where the frozen MAE forecasts worst (DMMV's core motivation), this directly compensates for the pretrained backbone's weakness. Qualitatively, the spectral branch yields smoother images without line structure (trend-dominated signal), while the structural branch yields vertical patterns for 1D ordering (Fig. 4). The branches thus capture different aspects, so fusion is best (single branch: +5.04% MSE on ETT).

Thus the spectral gap has in fact been accounted for in our model and yields good results, and SMA reduces the spectral gap.

The premise is that a pre-trained MAE works better when its input has natural-image spectral statistics, but this is never tested.

We addressed the relationship between PSS and forecasting accuracy in App. A.5: PSS and MSE are not monotonically related. PSS is rather a channel for observing and analysing a feature: a higher PSS means less high-frequency content, i.e. a weaker periodic / noise-like component and a stronger trend component, and a lower PSS means the opposite. Our SMA raises PSS only limitedly, i.e. it strengthens the trend-related features, but this exactly compensates for the pretrained MAE's own weakness in forecasting the trend. Blindly pushing PSS towards the natural-image value would instead destroy the structure of the time series itself.

...and with lambda = 0.05 alpha moves only slightly (e.g., ETTh1 from 1.37 to 1.48). The paper's own analysis finds the best MSE at the smallest alpha. It is therefore unclear whether the "spectral gap" is a real cause of error or just a descriptive statistic.

The value λ=0.05\lambda=0.05 is chosen empirically: a larger λ\lambda overwhelms the forecasting loss and distorts the series structure, while a smaller λ\lambda leaves the gap unregularised. Hence the resulting shift in PSS(α\alpha) is intentionally modest. A moderate PSS increase strengthens trend features. PSS is therefore a channel for observing and analysing the data distribution rather than a training objective, revealing how trend, periodicity and noise-like features are distributed.

On the smallest-α\alpha point, we agree the original design was flawed and add a periodicity sweep (ETTh1):

Periodicity1218242830324864
α\alpha0.921.151.341.682.432.622.852.86
MSE0.3870.4390.3430.4550.4770.4510.3910.444

As the table shows, the best MSE is not at the smallest α\alpha: α=0.92\alpha=0.92 (periodicity 12) gives MSE 0.387, while the best MSE 0.343 is at α=1.34\alpha=1.34 (periodicity 24, the default). Hence α\alpha and MSE are not monotonically related: the spectral gap is for observation and analysis, not a causal explanation of the error.

The text-vs-image comparison (alpha for ASCII-rendered text) also does not say anything about LLM forecasters, which do not consume rendered images.

As Table 1 shows, LVM-based forecasters (SSDA, DMMV, VisionTS) generally outperform LLM-based ones (Time-LLM, GPT4TS), and our spectral analysis offers a frequency-domain explanation. We include text precisely because it is the LLM forecaster's input modality; text is spectrally the most distant of the three. This distance stems from a fundamental modeling difference: a text-native LLM tokenises continuous values into discrete symbols, breaking numerical continuity, while rendering series as images converts trends and periodicity into gradients and textures the vision encoder already models. Text and images are thus not interchangeable for a pretrained backbone.

AuthorsOpenReview ↗

2. Clarifying the temporal indexing mechanism of TGA

We thank the reviewer for raising this question. A detailed response is given below.

Each patch corresponds to a group of samples rather than a single sample. Before resizing, each column contains one complete, contiguous period, and each row marks a position within that period. Resizing merely stretches this correspondence proportionally onto the pixel grid. A patch therefore covers a rectangular block of "phase interval by period interval", and near the boundary it may span several columns. For example, a patch near the top of the image covers samples near the beginning of each period it spans.

MAE reads patches in row-major order by default (left to right, then down), which conflicts with a layout where time advances down the columns. TGA instead reads the 14×1414\times14 patch grid from top to bottom, then moves to the next column on the right. Thus, patch index ii (counting from zero) refers to row i mod 14i\bmod 14 and column ⌊i/14⌋\lfloor i/14\rfloor: the column tells us which periods the patch covers, and the row tells us which part of those periods it covers. In this way, the index gives each patch an approximate location in the original series.

We do not overstate this: it is a relatively coarse-grained temporal positional encoding, giving an approximate rather than exact sample index. What it does restore is the correct order of the series — adjacent indices correspond to adjacent phases, and crossing columns corresponds to crossing periods, so the patch read-out order is strictly aligned with the original temporal order. The ablation study and the reconstruction visualization of Fig. 4 confirm that this recovery is effective.

AuthorsOpenReview ↗

3. Corrected ablation labels and Table 1 averages, and a new ensemble control

The labels of the two ablation variants are swapped between Table 3 and Table 11, so it cannot be read which branch matters.

We thank the reviewer for spotting this error and apologise for the oversight. We accidentally swapped the labels of the two ablation variants and have corrected their order in Tables 3 and 11. All numerical results remain unchanged.

The corrected Table 3 is shown below. Each entry reports MSE / MAE. A negative value in a Drop row indicates higher error than the full model. The final column averages the values in each row across the four datasets.

Method (↓), Metric (→)ETTh1ETTh2ETTm1ETTm2Avg
MSE MAEMSE MAEMSE MAEMSE MAEMSE MAE
SSDA (full)0.387 0.4060.327 0.3760.332 0.3630.249 0.3110.324 0.364
(1) w/o structural branch (SMA only)0.403 0.4170.332 0.3780.362 0.3740.288 0.3310.346 0.375
Drop−4.13% −2.71%−1.53% −0.53%−9.04% −3.03%−15.66% −6.43%−7.59% −3.18%
(2) w/o spectral branch (SG-LoRA only)0.391 0.4080.332 0.3780.342 0.3670.260 0.3180.331 0.368
Drop−1.03% −0.49%−1.53% −0.53%−3.01% −1.10%−4.42% −2.25%−2.50% −1.09%
(3) w/o spectral branch & TGA (LoRA only)0.462 0.4380.396 0.4100.353 0.3690.272 0.3230.371 0.385
Drop−19.38% −7.88%−21.10% −9.04%−6.33% −1.65%−9.24% −3.86%−14.01% −5.61%

As the table shows, both branches are necessary — removing either degrades performance on all four datasets, which supports the complementary-gap thesis. Removing the structural branch (SG-LoRA + TGA) costs −7.59% MSE / −3.18% MAE on average over the four ETT datasets, whereas removing the spectral branch (SMA) costs a smaller but still consistent −2.50% MSE / −1.09% MAE. The structural branch is therefore the larger contributor.

Under the Table 11 labels, the spectral-only model is worse than plain VisionTS on ETTm1 and ETTm2.

This is indeed the case, but it does not affect SSDA's SOTA standing. SSDA is by design a two-branch complementary model: the spectral branch strengthens trend features while the structural branch strengthens 1D temporal-structure features, and only their fusion reaches SOTA. The spectral-branch result here is just a single-branch ablation outcome, so being weaker than VisionTS on some datasets is expected. In fact, the full SSDA outperforms VisionTS on all seven datasets (averaged MSE in Table 1): by 4.91% / 6.84% / 3.49% / 6.74% on ETTh1 / ETTh2 / ETTm1 / ETTm2, and by 3.56% / 1.89% / 1.30% on Weather / Electricity / Traffic, while taking 48 first places against VisionTS's 6, confirming the SOTA effect of the dual-branch fusion.

In addition, SSDA mixes a frozen-MAE forecast with a LoRA-tuned-MAE forecast, which is a two-model ensemble; the natural control (frozen VisionTS + LoRA-VisionTS mixed with beta, without SMA and TGA) is missing.

As requested by the reviewer, we have run this control: the same two-branch ensemble (frozen MAE + LoRA-MAE, mixed by a single learned scalar), but with neither SMA nor TGA. To highlight the source of the gain, the table below keeps only the averaged results (MSE / MAE) for SSDA, VisionTS, and this control (VisionTS from Table 1 of the paper).

MethodETTh1ETTh2ETTm1ETTm2Avg
MSE MAEMSE MAEMSE MAEMSE MAEMSE MAE
SSDA (full)0.387 0.4060.327 0.3760.332 0.3630.249 0.3110.324 0.364
VisionTS (frozen MAE)0.407 0.4190.351 0.3860.344 0.3730.267 0.3270.342 0.376
Drop−5.17% −3.20%−7.34% −2.66%−3.61% −2.75%−7.23% −5.14%−5.56% −3.30%
frozen MAE + LoRA-MAE0.400 0.4250.337 0.3850.340 0.3730.254 0.3180.333 0.375
Drop−3.36% −4.68%−3.06% −2.39%−2.41% −2.75%−2.01% −2.25%−2.78% −3.02%

Comparing this control with the full SSDA model, we find that SSDA outperforms both the frozen MAE + LoRA-MAE mixed by β control and VisionTS on the four-dataset average. Since the control shares the identical ensemble mechanism and differs only in the presence of SMA and TGA, this gain is attributable to our two proposed components rather than to the ensemble itself.

Some averages in Table 1 also do not match their own per-horizon values (they match Table 11 instead), which suggests the tables were not regenerated from one run.

We thank the reviewer for such a careful cross-check of our experimental data. We went through Tables 1 and 11 entry by entry and found only two discrepancies: the MAE at ETTh1 H=336 (0.430 / 0.420), where 0.430 is a typo and the correct value is 0.420; and ETTm2 H=96 (0.162/0.249 vs. 0.168/0.256), where different experimental results may have been used, but the impact is minor — both of these results remain at SOTA level for SSDA. We will correct them in the paper.

Reviewer_UWAnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SSDA adapts a pretrained vision model for time series represented as images. One branch changes the image magnitude spectrum while retaining phase. The other adds temporal position embeddings and uses LoRA to adapt attention. A learned weight combines their forecasts. Experiments cover seven forecasting benchmarks, including full-shot and few-shot settings.

Critical strengths
  1. The paper identifies two concrete problems in converting time series into images: different frequency statistics and disrupted temporal order. The two branches address these problems separately.
  2. The experiments cover several datasets and forecast horizons, with additional results for limited training data and component ablations.
Critical weaknesses
  1. The temporal mapping in TGA is unclear. Position embeddings use patch indices, but the series has already been folded into a 2D image and resized. How does each patch index map back to the original time axis?

  2. The evidence for spectral alignment is limited. SMA is not explicitly trained to match natural-image spectra, and Appendix A.5 notes that spectral slope need not track forecasting accuracy. Could you compare SMA with a simple learnable image transformation under a similar parameter budget?

  3. The branch ablation results conflict between Tables 3 and 11. For ETTm2, Table 3 gives MSEs of 0.288 and 0.260 after removing the spectral and structural branches, respectively, while Table 11 reverses them. The same reversal appears for ETTh1 and ETTm1. Please correct the labels and check the conclusion about which branch contributes more.

Questions for authors

Please see weaknesses.

5 comments
AuthorsOpenReview ↗

1. Temporal mapping in TGA

We thank the reviewer for raising this question. A detailed response is given below.

Response

The temporal mapping in TGA is unclear.

Each patch corresponds to a group of samples rather than a single sample. Before resizing, each column contains one complete, contiguous period, and each row marks a position within that period. Resizing merely stretches this correspondence proportionally onto the pixel grid. A patch therefore covers a rectangular block of "phase interval by period interval", and near the boundary it may span several columns. For example, a patch near the top of the image covers samples near the beginning of each period it spans.

MAE reads patches in row-major order by default (left to right, then down), which conflicts with a layout where time advances down the columns. TGA instead reads the 14×1414\times14 patch grid from top to bottom, then moves to the next column on the right. Thus, patch index ii (counting from zero) refers to row i mod 14i\bmod 14 and column ⌊i/14⌋\lfloor i/14\rfloor: the column sets the periods, the row sets the phase, giving each patch an approximate location in the original series.

We do not overstate this: it is a relatively coarse-grained temporal positional encoding, giving an approximate rather than exact sample index. What it does restore is the correct order of the series -- adjacent indices correspond to adjacent phases, and crossing columns corresponds to crossing periods, so the patch read-out order is strictly aligned with the original temporal order. The ablation study and the reconstruction visualisation of Fig. 4 confirm that this recovery is effective.

How does each patch index map back to the original time axis?

The patch-to-time correspondence is given by the rendering and inverse-rendering. A sample tt falls on period p=⌊t/P⌋p=\lfloor t/P\rfloor and phase f=t mod Pf=t\bmod P, and resizing only stretches this proportionally, so the pixel position of tt can be computed directly. Conversely, a patch's row marks a phase interval and its column marks a period interval -- together they give its approximate location in the original series.

AuthorsOpenReview ↗

2. Spectral effect of SMA and control experiment

We thank the reviewer for raising this question. A detailed response is given below.

Response

SMA is not explicitly trained to match natural-image spectra, and Appendix A.5 notes that spectral slope need not track forecasting accuracy.

We agree with the reviewer's points. SMA is not trained with an explicit natural-image spectral-matching loss; it is trained end-to-end for forecasting, to learn the latent characteristics of the series in the frequency domain. PSS and MSE are not monotonically related either (App. A.5). The rise in PSS caused by SMA is a consequence of improving forecasting accuracy.

Even so, SMA's design remains consistent with our motivation. Methodologically, once time-series or image data is converted to the frequency domain, it is represented by magnitude and phase. Changing the magnitude changes the texture / frequency statistics, whereas changing the phase would very likely destroy the data's own characteristics. SMA therefore applies its learnable transformation to the magnitude.

Empirically, the spectral branch does reshape the frequency-domain characteristics of the series: after SMA training, the PSS of the rendered images increases on every dataset, i.e. the periodic / noise-like component is weakened and the trend component is strengthened -- and the trend is precisely what a frozen natural-image MAE handles poorly, which is the core problem DMMV sets out to address. Qualitatively (Fig. 4), the spectral branch produces smoother reconstructions without pronounced line structure, matching a trend-dominated signal, whereas the structural branch produces vertical patterns consistent with 1D temporal ordering. The two branches capture different aspects of the series, which is why combining them works best.

Could you compare SMA with a simple learnable image transformation under a similar parameter budget?

Following the reviewer's suggestion, we compared SMA with two CNNs that operate directly on the image pixels. The Spatial CNN applies the same convolutional layers, dropout, and residual coefficient as SMA, but directly on the time-domain input rather than on the frequency-domain magnitude. The Conventional conv removes dropout and uses a learnable sigmoid gate for the residual. We used the same structural branch in all variants.

The table reports MSE averaged over forecasting horizons of 96, 192, 336, and 720. Each ΔMSE is the percentage increase relative to SMA on the same dataset.

VariantETTh1 MSEΔMSEETTh2 MSEΔMSEETTm1 MSEΔMSEETTm2 MSEΔMSE
SMA0.387—0.327—0.332—0.249—
Spatial CNN0.393+1.55%0.335+2.45%0.338+1.81%0.252+1.20%
Conventional conv0.405+4.65%0.343+4.89%0.339+2.11%0.258+3.61%

SMA achieved the lowest MSE on all four datasets. The most direct comparison is with the Spatial CNN, which uses the same architecture and parameter count. Its MSE was at least 1.20% higher than SMA's on average across datasets. This result supports applying the transformation to the Fourier magnitude for forecasting.

AuthorsOpenReview ↗

3. Correction of the branch-ablation labels in Tables 3 and 11

We thank the reviewer for spotting this error and apologise for the oversight. We accidentally swapped the labels of the two ablation variants and have corrected their order in Tables 3 and 11. All numerical results remain unchanged.

The corrected Table 3 is shown below. Each entry reports MSE / MAE. A negative value in a "Drop" row indicates higher error than the full model. The final column averages the values in each row across the four datasets.

Method (↓), Metric (→)ETTh1ETTh2ETTm1ETTm2Avg
MSE MAEMSE MAEMSE MAEMSE MAEMSE MAE
SSDA (full)0.387 0.4060.327 0.3760.332 0.3630.249 0.3110.324 0.364
(1) w/o structural branch (SMA only)0.403 0.4170.332 0.3780.362 0.3740.288 0.3310.346 0.375
Drop−4.13% −2.71%−1.53% −0.53%−9.04% −3.03%−15.66% −6.43%−7.59% −3.18%
(2) w/o spectral branch (SG-LoRA only)0.391 0.4080.332 0.3780.342 0.3670.260 0.3180.331 0.368
Drop−1.03% −0.49%−1.53% −0.53%−3.01% −1.10%−4.42% −2.25%−2.50% −1.09%
(3) w/o spectral branch & TGA (LoRA only)0.462 0.4380.396 0.4100.353 0.3690.272 0.3230.371 0.385
Drop−19.38% −7.88%−21.10% −9.04%−6.33% −1.65%−9.24% −3.86%−14.01% −5.61%

Taken together, the table confirms that both branches are necessary: removing either one degrades performance on all four datasets, which supports our complementary-gap thesis. The two branches do not contribute equally, however. Removing the structural branch (SG-LoRA + TGA) incurs a sizable average cost of −7.59% MSE / −3.18% MAE across the four ETT datasets, whereas removing the spectral branch (SMA) incurs a smaller yet consistent cost of −2.50% MSE / −1.09% MAE. The structural branch is therefore the larger contributor.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.