acceptodds
Under review as a conference paper at ICLR 2027

MM-TS: Channel-Structured Vision-Language Modeling for Multivariate Time Series Forecasting

Abstract

Multivariate time-series forecasting requires modeling temporal dynamics and evolving cross-channel dependencies. Recent LLM- and VLM-based forecasters adapt numerical sequences to pretrained foundation models through textualization, patch reprogramming, or visual reconstruction. However, adapting inputs to a pretrained backbone does not by itself provide explicit channel-relation guidance for temporal-token attention. We propose MM-TS, a channel-structured vision-language framework for multivariate time-series forecasting. MM-TS reuses global and local channel relations derived from each input window in two complementary ways: as visual tokens and as signed biases on temporal-token attention logits. Relations from the full-window magnitude spectrum form an image, while relations from successive frequency-axis patches form the video frames. A pretrained vision-language backbone jointly processes these visual tokens, time-series tokens, and textual prompts. To preserve channel-pair identity, local relations occupy diagonal patch blocks in the attention-bias matrix, whereas the global relation is broadcast to off-diagonal blocks. This dual use aligns each relation entry with the attention edges of its channel pair, enabling channel structure to guide information routing rather than remain an input-only representation. Evaluation covers eight standard benchmarks and ten additional TIME datasets spanning varied variables and sampling resolutions. Against nine baselines, MM-TS achieves the lowest horizon-averaged MSE on 14 of these 18 datasets. It also remains competitive with limited training data and achieves the lowest average MSE in five of six evaluated ETT transfer directions without target-domain weight updates. Controlled ablations support the complementary contributions of the two paths and the importance of channel-pair correspondence and input-dependent relations. Code is available at https://anonymous.4open.science/r/MM-TS-DACB/.

Primary area:
learning on time series and dynamical systems
Keywords:
Multivariate Time Series Forecasting, Vision-Language Models, Cross-Channel Dependency Modeling, Structured Attention Bias, Foundation Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_buZEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents MM-TS, a channel structured vision-language model for multivariate time series forecasting. It reuses time-series-derived relation matrices for both vision token construction and attention guidance. In addition, it proposes a structured attention bias to preserve channel-pair identity: local spectral relation and global relation.
With this data preprocessing, all input modalities (text, image, video, numerical) are input to the Qwen3 VL Decoder for multivariate prediction.

Experiments are conducted on eight standard benchmarks and ten additional TIME datasets. The proposed method MM-TS achieves competitive or better performance on both full and limited training data settings.

Critical strengths

This paper models the channel structurals and injects them into the attention bias, providing reasonable design for multivariate time-series forecasting.

  • Multi-modal input are extracted and preprocessed for the multivariate time-series forecasting task, including text, patch to image, patch to video, and original numerical time-series.
  • The image and video inputs capture cross-channel relation to model the multivariate relationship.
  • Structued bias are injected into the attention module of Qwen decoder for prediction.

Better or competitive results are achieved by this method + the strong Qwen3 backbone.

  • With the proposed data augmentation and attention bias injection pipeline on top of the strong Qwen3 backbone, it achives better or competitive results on 18 datasets.
Critical weaknesses

The design is simple and relatively straightforward, introducing limited insights and novelty to the time-series and multivariate time-series task.

  • Time-LLM already combines text inputs with time-series data and verifies the effectiveness.
  • Existing vison-based time-series forecasting models such as VisionTS leverage patched image information for the problem.
  • Video information is relatively new in this setting, but extending from image to video is straightforward and simple. There exists video foundation models that can be leveraged for time-series forecasting.
  • DMMV [R1] incorporates the vision information + original numerical time-series for the forecasting problem.

Although better or competitive results are achieved, it is unclear whether the improvements come from model design or stronger backbone.

  • Existing methods use relatively lighter backbone with smaller parameters, while this paper uses Qwen3 VL as a strong backbone. Qwen3 VL has much larger parameters and memory consumption, thus leading to slower inference. This can be seen from Appendix F.

This is unfair comparison with existing method results.
From Table 3, without the module designs, the bare backbone already achieves strong performance comparable or beat existing methods. Therefore, the better performance of this paper might come from the backbone not model design.

  • DMMV [R1] as a related and strong baseline method is not compared in this paper.

[R1] Shen, C., Yu, W., Zhao, Z., Song, D., Cheng, W., Chen, H., & Ni, J. (2026). Multi-modal view enhanced large vision models for long-term time series forecasting. Advances in Neural Information Processing Systems, 38, 93269-93296.

Questions for authors

The questions mainly lie in the novelty of the proposed method, as well as fair comparison with existing methods. Details are listed in above Weaknesses.

2 comments

1. Inference speed is indeed a limitation. However, most of the time is spent converting the time series into images and video, rather than on Qwen3 inference itself. I have not yet found a good solution to this problem.

2. I acknowledge that part of the strong performance comes from Qwen3-VL, which is also why we chose it. We have added further backbone comparison experiments and analysis in Table 4. Your concern that choosing Qwen3-VL may make comparisons with other models unfair is reasonable. However, many recent baselines cannot be adapted to Qwen3-VL, and requiring all comparisons to use Qwen3 would also be unfair. In addition, we have examined papers with publicly available code from conferences such as ICML and found comparisons involving different backbone models, such as GPT-2 and Llama 3. These are not isolated cases. I am therefore currently unsure how best to address this issue.

3. I will read the papers you recommended more carefully.

Reviewer_buZEOpenReview ↗

Thank you for the rebuttal

The authors replied with "not yet found a good solution", "unsure how best to address this issue", and "will read the papers you recommended more carefully".

This indicates that the response does not address the core concerns. Therefore, I will keep my score for this paper.

Reviewer_T9pFOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces the MM-TS architecture for multivariate time-series forecasting. MM-TS is based on an LLM that receives text as input and time series as image, video, and time-series soft tokens. Time series tokens are produced by projecting raw values onto the LLM (Qwen3 VL)embedding space using an MLP. Image and video tokens are obtained by 1) computing windowed FFT, 2) (perhaps from the frequency bins) computing feature maps of DTW, covariance, and Pearson correlation for each pair of channels, 3) converting them into RGB format, and 4) passing through the Qwen vision encoder. The reported benchmark results show MM-TS as a strong method. However, the empirical evaluation has several critical flaws pertaining to baselines, evaluation procedure, reported numbers, and, in general, methodology. Moreover, the motivation of this paper is very fuzzy and never convincingly explained; the central research question investigated in this work is hard to find. Furthermore, the quality of writing is subpar; there are numerous examples of jargon and difficult-to-parse sentences. Finally, the positioning to related work is very limited, and many relevant concurring works have been left out. For these reasons, I recommend rejection of this paper.

Critical strengths
  1. Code is available online.
  2. Multimodal time series forecasting and alignment of time series and LLMs is an important topic.
Critical weaknesses
  1. Evaluation

    1. Table 1, standard long-horizon benchmarks. There is a disparity in reported baseline numbers in this paper as compared to the numbers reported by the papers introducing these baselines. For instance, MM-TS reports .190 on Electricity and .192 for iTransformer, yet in the iTransformer paper this number is .178. Moreover, the authors cite T3Time (Chowdhury et al. 2026), but do not compare to it; importantly, T3Time reports iTransformer as .178 and their method as .190 for Electricity.
    2. Limited evaluation. The term “standard benchmarks” does not suffice to establish a convincing result and prove that MM-TS works well. The results are averaged across forecasting horizons but not across varying context lengths, randomness (e.g., seed), and do not report either standard deviation or standard error (or confidence intervals).
    3. The most recent multivariate forecasting baseline is from 2024, which in this rapidly developing field is not acceptable. As mentioned above, the authors cite T3Time (Chowdhury et al. 2026), but do not compare to it. I do not want to point to specific papers that authors should have compared to, but there are many papers from 2025 and 2026 with >100 citations published at major AI conferences that introduce multivariate or LLM-based forecasting architectures; comparison with a selection of them is necessary. Moreover, as benchmarks like Context is Key (Williams et al. 2024) show, directly prompting state-of-the art large LLMs provides state-of-the-art performance. Therefore, it would strengthen the paper to see directly prompted LLMs as baselines, especially because MM-TS is quite expensive at inference time.
    4. TimesFM uses a dated 1.0 checkpoint from 2023, while the same Hugging Face interface provides access to TimesFM 3.0 (2026) and TimesFM 2.5 (2025). I do not see why a comparison to TimesFM 1.0 would be relevant if newer checkpoints exist and can be seamlessly integrated into the same evaluation pipeline. Especially since TimesFM seems to represent TSFM-class models in this paper. Notably, most contemporary TSFMs are multivariate; thus, one could seek comparison with a selection of a few representatives (e.g., TimesFM 3, Chronos 2, Toto 2).
    5. The baseline evaluation setup is not explained. Have they been trained from scratch, or do they use some public checkpoints? What hyperparameters have been used? MM-TS temporal patches have length 16 and stride 8. Are these hyperparameters arbitrarily chosen, or have they been tuned? What would have happened if they were different? What’s the sensitivity of MM-TS to these hyperparameters? How about LoRA ranks?
  2. Motivation is not well-supported, and positioning w.r.t. related work is weak.

    1. Authors claim multiple times that existing methods do not allow for cross-channel interactions and that these interactions are absorbed by tokens or patches. I do not understand the second part of these claims, because tokens and patches are usually formed within a channel, concatenated, and then provided to the LLM in a prompt; thus each channel gets multiple patches/tokens, and the entire input consists of multiple channels. There is no architectural limitation in LLMs/Transformers that hampers learning cross-channel interactions.
    2. Related work has no positioning w.r.t. multivariate forecasting methods which are not LLM-based and some of which appear later in the benchmarks. The sentence about TSFMs is detached from the flow of the section, and the juxtaposition with MM-TS that follows completely misses relating MM-TS to TSFMs and highlighting relevant differences. The coverage of related LLM-based methods is minimal, and the explanation for why they ignore the cross-channel interactions is not convincing.
  3. Writing quality is not up to ICLR-level standards. Until late in Section 3, I still didn’t know what the authors are trying to propose in this paper. While Figure 1 is helpful, both the introduction and related work are difficult to parse, and the writing quality is low. Furthermore, in Section 3 there are just too many issues with writing and clarity. I cannot follow the method because the authors use many synonyms and jargon that have not been explained in the text. A few examples: “frequency-axis patches from the video frames”, “channel-pair identity”, “Cached spectral relations also guide temporal-token attention”; “cached relation matrices”, “patch-major order”, “Decoder-layer indices are zero-based throughout”, “PC multimodal tokens”.

    Since motivation and clarity in the research question are limited in this paper, the methods section is the most relevant one, yet it’s so convoluted that decoding what the method actually does is very hard, and a lot of details are missing or left unexplained.

  4. Computational cost, training time, and inference memory of MM-TS are higher than those of all evaluated baselines (Appendix F). One of the main components of MM-TS seems to be the DTW feature map computed for all pairs of channels. This is very expensive: O(P2C2)O(P^2C^2) (notation: P - number of patches per channel, C - channels)

Improper citation: Haoyi Zhou. Electricity transformer dataset (ETDataset). https://github.com/ zhouhaoyi/ETDataset, n.d. Accessed September 25, 2026.

The link in this citation leads to a github page that provides a correct copyable citation key and asks for proper attribution: “If you use this dataset please cite the work Informer @ AAAI2021 Best Paper Award ”: @inproceedings{haoyietal-informer-2021, author = {Haoyi Zhou and Shanghang Zhang and Jieqi Peng and Shuai Zhang and Jianxin Li and Hui Xiong and Wancai Zhang}, title = {Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting}, booktitle = {The Thirty-Fifth {AAAI} Conference on Artificial Intelligence, {AAAI} 2021, Virtual Conference}, volume = {35}, number = {12}, pages = {11106--11115}, publisher = {{AAAI} Press}, year = {2021}, }

Questions for authors
  1. See questions in Weaknesses 1.5
  2. What is MM-TS doing that within-channel patching with concatenation across channel and projection to LLM embeddings cannot do?
3 comments

May I ask whether you are a human reviewer? If so, I would be very happy to discuss the paper with you.

I will not engage with such comment.

I apologize for my previous question. The issues you raised are indeed valuable, but addressing them properly would require extensive experiments, which may not be feasible in the short term. Nevertheless, I intend to continue working on this research direction. If you don't mind, could you recommend two exemplary papers that meet the standards you mentioned above? I would greatly appreciate the opportunity to learn from them. Thank you for your guidance!

Reviewer_b1u3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MM-TS for multivariate time series forecasting. The idea is to first convert the time series info into text, image, numerical, and video information. Then they are fed into Qwen3 for a continuous training. They feed them as embedding layers and also as bias to the attention layers. The method is tested on 18 datasets, with additional few-shot, transfer, and ablation experiments.

Critical strengths
  1. The paper provides a concrete way to connect channel relations with attention between the corresponding channel pairs. The relation matrices are used both as visual inputs and as attention biases.

  2. The experimental coverage is broad. The authors evaluate 18 datasets with five seeds and include useful controls, such as removing visual tokens, removing attention bias, permuting channel correspondence, and using previous-window relations.

  3. The method improves forecasting on many datasets. Some improvements, such as ETTh2 and Coastal T-S, are substantial compared with the best reported baselines.

Critical weaknesses
  1. The novelty compared with existing multimodal forecasting methods is not sufficiently clear. Time-VLM already combines numerical time series with visual and textual representations, and VisionTS reformulates forecasting as image reconstruction. Converting multivariate sequences into video also has earlier examples, such as Deep Video Prediction for Time Series Forecasting (2021). I understand that these methods do not use the exact design proposed here. The more specific contribution of MM-TS appears to be constructing spectral channel-relation images and video frames, and reusing the same relations as attention biases.

  2. The correspondence between frequency patches and temporal patches is not well justified. Section 3.3 constructs video frames by cutting the full-window FFT magnitude along the frequency axis. However, Equation (6) assigns the relation from frequency patch p to attention within temporal patch p. These two indices have different meanings. A frequency band describes components across the whole observed window, and does not naturally correspond to one particular time segment. I am not sure why this assignment is appropriate. The channel-permutation experiment checks channel-pair identity, but does not test this frequency-to-time mapping.

  3. It is unclear how much the visual representation contributes beyond providing the same numerical relation features to the backbone. Time-VLM already combines temporal, visual, and textual representations, and adding structural bias to attention also has existing precedents. The more specific contribution here is their combination using channel-relation matrices. Removing visual tokens shows that this input path helps, but what if the same matrices are directly mapped into prefix tokens by a small MLP? Resetting the visual-language projector and replacing the backbone with DINOv2 do not directly answer this question. In my opinion, this comparison is needed to justify converting numerical matrices into RGB images and video.

  4. The image layout introduces an additional assumption about channel ordering. Rows and columns represent variables, rather than physical locations, so neighboring pixels may not have a meaningful spatial relationship. Changing the order of variables changes the visual patterns even when the underlying forecasting problem remains the same. The current permuted-bias control deliberately breaks channel correspondence. It does not test a consistent permutation of numerical channels, relation-image rows and columns, and the corresponding channel identifiers. This matters when claiming that pretrained visual features are useful for these synthetic images.

  5. The accuracy gain needs to be considered together with the model cost. For example, ETTh1 average MSE improves from 0.437 to 0.434, while Table 21 reports about 2.06 billion total parameters and 25.8 ms inference latency per sample, compared with 1.55 ms for the ordinary iTransformer. The paper reports these costs, which is useful. Still, I am not convinced that the additional visual-language machinery is justified over simpler relation-enhanced forecasting models. The iTransformer control includes only the global bias, so it does not fully test whether a smaller model can benefit from the proposed local and global structure.

Questions for authors
  1. Why should frequency patch p guide attention specifically within temporal patch p? Could the authors compare this with training under alternative frequency-to-time assignments, a pooled local relation shared across temporal patches, or relations computed from the actual temporal patches?

  2. Could the authors replace the visual encoder with a small numerical encoder that maps the same relation matrices into prefix tokens, while keeping the Qwen backbone, attention bias, and token budget comparable?

  3. Does performance remain stable under a consistent reordering of channels across all input paths, with predictions mapped back to the original order? This would help separate useful channel relations from dependence on a particular image layout.

  4. Could the full local/global bias design be evaluated in a smaller patch-based Transformer? The current iTransformer comparison tests only the global component, leaving the benefit of the full structural design outside the VLM setting unclear.

1 comment

First, thank you for taking the time to review our paper. It has been a long time since I last received feedback from a human reviewer. I would like to focus our discussion mainly on the Weaknesses section:

1. The main novelty lies in reusing multimodal information as attention biases within the VLM. The insufficient presentation of this novelty may be due to my writing, or perhaps I have not read enough of the literature in this area. I will spend more time reflecting on where the issue lies.

2. I agree with your concern. I need to conduct further repeated experiments across multiple datasets to reconsider whether the Fourier transform step is actually necessary.

3. We have conducted some visual ablation experiments, and I will also try adding the experiment you suggested. My exploration of backbone models has so far been limited to lightweight models such as smaller Qwen3-VL variants, GPT-2, and standard Transformers. One important limitation of our model is that its inference cost grows quadratically with the number of channels, O(C2)\mathcal{O}(C^2). Finding ways to make inference faster is worth further exploration.

4. I do not fully understand this point yet. I will study it further. Thank you for your suggestion.

5. I have addressed this in my response to point 3.

I have been working on this project for a year and a half, with much of that time spent conducting and testing various experiments. There are still shortcomings in the writing and the presentation of its novelty. I am glad that there is still considerable room for improvement, and I will continue to refine this work. Thank you again.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.