acceptodds
Under review as a conference paper at ICLR 2027

Vision Pretraining for Time-Series Forecasting Depends on More Than Undirected Grid Topology

Abstract

Vision models can forecast time series by treating numerical sequences as masked images, but why natural-image pretraining transfers remains unclear. We test whether undirected grid structure explains this benefit. Phase, cycle, and joint reflections preserve observed values and undirected adjacency exactly while changing how temporal structure is embedded in the visual coordinate system. Across 11 datasets, 71 series, 2,272 forecast origins, and three masked-image checkpoints, reflections consistently reduce pretrained-weight advantage. The effect remains under architecture-matched random controls and alternative metrics including MAE, MASE, and sMAPE. A post-hoc loss decomposition shows that reflections primarily increase pretrained-model losses while leaving matched random-model losses nearly unchanged. These results show that undirected topology alone cannot explain cross-modal transfer from vision pretraining to time-series forecasting; the roles of direction, absolute position, and boundary anchoring remain unresolved.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
time-series forecasting, vision pretraining, cross-modal transfer, representation learning, pretrained models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ziQVOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper conducts an experimental comparisons to investigate the effectiveness of leveraging vision model pretrained on image data to apply into the time series forecasting tasks. It visits the numerical data representation in images as usually undirected topology, and reports that the undirected topology setting is insufficient for transferring vision model to time series tasks and justifies it via phase, cycle, and both reflections. It then conducts some numerical experiments to further explain such an observation.

Critical strengths

Well, this work visits a very specific potential issue of leveraging pre-trained vision-based models to conduct time series tasks. Its problem description is clear.

Critical weaknesses

It is more or less like proposing a problem with insufficient technical development, e.g., proposing new mechanisms, for addressing such a problem. It aims to prove the existence of such an issues. However, to many of the time-series foundational model researchers, this is quite some well known issues. Hence, this is more or less a technical analytical report rather than an original research article presenting new findings, new methods, or new results. Thus, I think it is off the standard of the bar of the top AI conference.

Questions for authors

Q1. Why this issue deserves a research space? What is the motivation or necessity of a discussion on leveraging Vision models for time series forecasting tasks? Maybe it is not a valuable topic we should explore unless we justify the significant advantages of Vision based models on conducting time series forecasting.

Q2. Besides this image data understanding or analysing issue of this vision model, it is more important to understand the clear advantages of leveraging vision representation against the numerical sequence values in time series modelling tasks. Some of the recent works have discussed some; yet, to me, not deep or decisive enough. If it is really effective to use image representation of time series, then the path of studying statistical numerical time series methods or even those LLMs leveraging numerical sequence will be ended in the end. Thus, the studying of vision based TS modeling needs a more stronger justifications in terms of theoretical perspective to win this competition.

Q3. This is more or less a positioning paper rather than a research article? I feel that it just presents the argument of the authors. However, I cannot see sufficient technological advancement in terms of TS forecasting methods.

Reviewer_76xvOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper investigates reasons that could explain the effectiveness of VisionTS, an approach that treats timeseries as images and uses pretrained MAE to predict missing values. It does so by examinig how a set of transformations over the grid projected timeseries affects the difference in log loss between a pretrained MAE and a random model using the same architecture. In doing so it establishes the effect of these transformation on the pretrained-random model advantage.

Critical strengths
  • The topic is generally interesting, the fact that a MAE pretrained on natural images can be used for timeseries prediction with better that random performance is on its own a topic worth investigation.
  • The proposed transformations are a proposition for probing the phenomenon of knowledge transfer between these separate domains between natural images and general TS data.
Critical weaknesses
  • "Because A compares pretrained and random models under the same transformation, a layout be- coming uniformly harder cannot explain the change." I disagree on this and believe this is a major limiation of this work. Precisely a random model has not developed any structural biases that get affected by transformation of the data, its weights are random anyway. On the other hand a pretrained model has developed these "learnings" and is affected by their corruption. This also shows in Fig.3 where the random model is practically unaffected by the transformations and all the diff comes from the pretrained checkpoint. This brings into question if teh main motivation of the difference metrics between prtrained and random used by this study holds any values since the random loss behaves more like a constant and could be excluded. In that case it is not the difficulty of the task itself that affects performance, since any rewiring of teh pretrained network would match the performance of the model on non transformed data, but the corruption of several data biases that transfer between images and general timeseries data.
  • Following up from the point above, the paper doesn't seem to investigate biases in image data that could transfer directly to image time series and how these are corrupted from uh, the transformations proposed. Since there is no real free lunch in ML there should be some properties of the natural image data that transfer to TS, the paper achknowledges this fact in the intro. What are these properties and how they are affected by the proposed transformations is not examined.
  • The paper doesnt produce a concrete takeout for the reader, there is not a clear conclusion or a proposition for a property that either explains or improves the VisonTS framework.
Questions for authors
  • what is the usefuleness of comparing the differences from random model since this appears to be unaffected by all transformations used? How would conclusion be different if you only checked Α instead of Δ_Α?
  • Fig.3. i found it surprising that cycle reflection systematically caused a lower drop than phase reflection. The reason is that phase reflection retains the temporal order of the TS (early is still left of late values) while cycle reflction places the start of the timeseries right before teh future. DO you have any intuitions why this is the case?
Reviewer_72vjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies why natural-image pretraining helps VisionTS-based forecasting, where a time series is rendered as an image and a frozen masked-image model reconstructs the future. Using value- and adjacency-preserving reflections of the rendered image and a pretrained-versus-random comparison, it shows that reflections consistently reduce the benefit of pretrained weights across datasets and checkpoints, so undirected grid topology alone cannot explain the transfer. A registered directed-geometry hypothesis fails, leaving the actual mechanism unresolved.

Critical strengths

S1. The studied problem is meaningful. Understanding what pretrained vision models actually exploit in rendered time series matters to a growing line of work (VisionTS, VisionTS++), and value-preserving reflections are a well-chosen control that isolates how the sequence is embedded in the visual coordinate system from which values and neighborhoods are present.

S1. The contribution is mostly empirical. The discovery/confirmation separation, frozen manifest, dataset-level bootstrap,metric robustness checks, and honest reporting of the failed primary hypothesis are good scientific practice and make the reported reflection effect credible.

Critical weaknesses

W1. The contribution is limited. The confirmed finding reduces to: a frozen MAE with fixed absolute position embeddings is sensitive to flipping the rendered image. Natural images are not vertically symmetric and MAE/Swin position embeddings are tied to orientation, so this is close to expected, and "undirected topology is sufficient" is not a hypothesis that prior work defended. The directed displacement fails and no replacement is identified. For a main-track paper this leaves the contribution limited.

W2. The architecture-matched random control adds little to the inference. A randomly initialized MAE produces near-uninformative reconstructions regardless of layout, so its loss is nearly layout-invariant by construction (Sec. 5.2 confirms random-loss changes within about ±0.02). Hence the advantage estimand is effectively the change in pretrained loss. The matched comparison does not separate "harder for any network" from "harder for pretrained weights" when the random network cannot exploit any structure. Relatedly, Table 6 shows the frozen visual models barely outperform Seasonal Naive, so the transfer benefit being dissected is small in absolute terms, which may limit the practical significance of the conclusions.

Questions for authors

Q1. How much of the reflection effect is due to position embeddings alone? If the effect largely disappears without position embeddings, the finding appears to be a statement about position embeddings rather than about visual coordinate systems broadly.

Q2. Why is a random initialization the right reference rather than Seasonal Naive or an in-domain supervised model?

Q3. How about the performance of fine-tuned vision models? This matters because the practical use of vision backbones for forecasting typically involves adaptation, and the conclusion that transfer "depends on the visual coordinate system" would be considerably stronger if it is confirmed with fine-tuning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.