acceptodds
Withdrawn from ICLR 2027

WorldTS: World Modeling for Multimodal Covariate-Aware Time Series Forecasting

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Time series forecasting is typically framed as learning a direct mapping from historical to future observations in the observation space. However, sequences of observations generally provide only a partial view of the dynamics of the underlying system, with future observations being shaped by latent dynamics. Recent latent-space forecasting methods thus achieve improved performance by predicting future observations from latent-space representations of historical observations rather than directly forecasting future observations in the observation space. Next, while future observations are also shaped by external factors, how to incorporate external, often multimodal, information into forecasting, so that it can shape latent-state formation and evolution directly, remains underexplored. We propose WorldTS, a world-modeling based forecasting framework that integrates multimodal covariates directly into the forecasting to further improve forecasting performance. Specifically, WorldTS employs a two-stage training strategy. First, it learns forecasting-relevant latent state dynamics conditioned on multimodal covariates, yielding encoded future states. Next, the learned state dynamics are frozen, and an observation decoder is trained to map the predicted future states back to future observations. Extensive experiments on 21 real-world datasets offer insight into WorldTS and its effectiveness.

Primary area:
learning on time series and dynamical systems
Keywords:
Time Series

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_8PaHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies time-series forecasting with multimodal covariates. The method largely follows the latent-space forecasting approach of LatentTSF: a shared encoder maps historical and ground-truth future sequences into latent states, the predictor learns to match the encoded future states, and a separately trained decoder maps the predicted states back to observations. WorldTS extends this framework by incorporating numerical, textual, and visual covariates. Historical numerical covariates are concatenated with historical states and fused through an MLP, while text and image features are incorporated through a residual CFA adapter. Known future numerical covariates are used to further refine the predicted states. The model is trained in two stages, first optimizing the encoders and predictor with L1, cosine similarity, and VICReg losses, and then freezing them to train the decoder. The authors evaluate WorldTS on 21 datasets covering numerical, text, and image covariates. They follow the evaluation settings of DAG, VoT, and UniCA for the respective tasks, with baseline results taken from these works and KITE. WorldTS performs best in most of the reported comparisons. The paper also includes ablation studies on the loss terms, model components, training strategy, and covariates, along with hyperparameter sensitivity analyses on Colbun, Climate, and MMSP.

Critical strengths

1.The experimental results are strong. WorldTS is evaluated on 21 datasets covering numerical, textual, and visual covariates, and achieves the best results in most comparisons. Its performance is fairly consistent across these settings, suggesting that the approach is not limited to a particular type of data.

2.The ablation studies provide some support for the design choices. In Table 4, removing covariates or switching to one-stage observation-space training leads to worse performance on all three representative datasets. While these experiments do not fully isolate the contribution of each component, they do show that covariate information and latent-space training are useful for forecasting. And paper writing is clear.

Critical weaknesses

1.The incremental contribution over LatentTSF remains unclear. WorldTS adopts a similar latent-state prediction framework, with changes to the training scheme and the addition of multimodal covariates. However, without a direct comparison to LatentTSF or a controlled comparison with simpler covariate-conditioning strategies, it is difficult to determine which changes drive the gains. The current ablations show that covariates help, but do not fully resolve this question.

2.The stability of the reported improvements is uncertain. Baseline results are taken from prior work, and the comparability of tuning budgets is unclear. Moreover, the results are reported for a single seed without standard deviations. Some gains are quite small, MSE 0.285 vs. 0.288 on Agriculture and 0.090 vs. 0.091 on Time-MMD Energy, making it difficult to assess their robustness.

Questions for authors

1.Were the hyperparameters selected using validation or test performance? Section 5.3 and Figure 4 discuss the best-performing configurations, but the data split used and its role in hyperparameter selection are unclear.

2.Could the authors clarify the key methodological differences from LatentTSF and explain its omission from the experiments? If feasible, a target-only comparison between LatentTSF and WorldTS without covariates on one or two representative datasets would help assess the changes to latent-state learning.

3.Could the authors report mean and standard deviation over multiple seeds for WorldTS on a few datasets with small performance margins, such as Agriculture, Time-MMD Energy, or Colbun?

Reviewer_AW8SOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes WorldTS, a multimodal time-series forecasting framework that instead of mapping historical observations to future forecasts directly, it encodes current data into a sequence of states, from which a predictor infers the future states. The decoder then makes forecasts based on the future states. The first stage of training involves tuning (1) an encoder per modality (time-series, text, and image) that encodes data into a shared state space, and (2) a predictor that maps the current state into the future state obtained from encoding ground-truth future data. The second stage then freezes these modules to proceed to training a decoder that reconstructs the data from the states. Experiments on unimodal and multimodal time-series benchmarks show that WorldTS achieves competitive results.

Critical strengths
  • [S1] Simple and intuitive methodology. The approach of mapping multimodal data to a shared latent space for future state prediction is an intuitive take for modeling multimodal temporal data. The architectural choices are also simple and easy to follow, making reproduction easy together with the released code.
  • [S2] Comprehensive experimentation. Empirical studies are done across a wide variety of datasets, including both numerical-only and multimodal (with image and text) time series forecasting setups. These are datasets widely studied in time-series forecasting literature, which allows easy comparison for practitioners.
Critical weaknesses
  • [W1] Weak novelty. My biggest concern lies in the novelty of this work. The concept of performing prediction in the latent state space instead of the observation space was already studied in LatentTSF [A], from which this work appears to borrow many features, such as the 2-stage training pipeline and the objective functions for stage 1 training. The main difference against LatentTSF is in incorporating multimodal covariates, but this component is not developed in much depth and also largely borrows existing work (VoT for text, UniCA for images, CFA for fusion). As such, the technical novelty is relatively thin.
  • [W2] Unclear connection to world models. The paper frequently frames the work as a "world model", but the presented method doesn't explicitly model the dynamics of the latents, which I think is the main role of world models [B]. Instead WorldTS is trained to model single step predictions, mapping short historical patches to immediate future sequence of patches, without a learned transition function that evolves states through time. This is essentially just lifting the predictions often done directly in the observation space to the latent space, and each latent state's are not connected through a dynamical model. This makes the "world model" framing a bit of an overstatement.
  • [W3] Empirical significance. In the experimental results, the margins between performances of WorldTS and other baselines are quite thin in many points, with no variance statistics reported. To better assess the statistical significance of the work, it would be best to provide results averaged across multiple seeds, alongside standard deviations.

[A] Yang et al., From Observations to States: Latent Time Series Forecasting. ICML 2026.
[B] Ha and Schmidhuber., Recurrent World Models Facilitate Policy Evolution. NIPS 2018.

Questions for authors
  • [Q1] For Figure 1(d), how are the "w/o Cov" points obtained? If they are obtained by training the model with Covariates, but then the Covariates are removed during inference, it seems this shift in latent distribution is trivially expected and the results are not that informative. If they come from a differently trained model, comparing the latent features won't be easy.
  • [Q2] After Stage 1 training, how can we ensure that the embeddings from the frozen encoder contains enough information to reconstruct the time series data? The objective functions in Stage 1 only consider (1) aligning state predictions and ground-truth future states and (2) preventing representation collapse, but they never really consider encoding the actual information. Is this something that is imposed naturally with the two objective functions and a large enough dataset?
  • [Q3] Related to Q2, what motivates the two-stage training pipeline, or why can't we keep the predictor and encoder tunable during Stage 2? Another possibility is to train everything jointly in a single stage with the latent and observation losses combined.
Reviewer_yUnROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper extends latent time series forecasting (LatentTSF), which was proposed on the idea that future observations are a partial view of the latent dynamics of an underlying system, to the setting where covariates are available. LatentTSF improved performance in standard time series forecasting, but it did not account for external factors. To address this, the authors design how external factors are incorporated and how the model is trained with them, and propose WorldTS. Experiments on 21 real-world datasets support the effectiveness of WorldTS.

Critical strengths

S1. The paper is well written and easy to read overall.

S2. The motivation is clear, and as presented, the performance in the tables looks very strong.

S3. Code for the experiments is provided, which increases the transparency of the experiments.

Critical weaknesses

W1. The work itself addresses the question of how to incorporate external factors into LatentTSF, and the method is essentially a LatentTSF-like architecture with a few added components, so the novelty does not feel large. I do not think this is something the authors can fix, so there is no need to respond to this point.

W2. It is hard to say that the experimental results demonstrate the effectiveness of WorldTS. Could the authors justify the following two points?

W2-1. I checked the baseline numbers in Table 2 against the VoT paper. They match the H=12 rows of VoT's Appendix G, but VoT reports averages over four horizons ({6, 8, 10, 12}, or {12, 24, 36, 48} for Energy and Health), while Table 2 reports only H=12. For Energy and Health, H=12 is the shortest horizon in that protocol. Why was a single horizon chosen? Can the authors report results on all four horizons as VoT does?

W2-2. Regarding Table 3, I checked UniCA. UniCA also reports Time-MMD results, and Time Series Foundation Model + UniCA is the strongest method there. Table 3 includes UniCA-adapted TSFMs as baselines on MMSP, but Table 2 has no TSFM-based baselines on Time-MMD. I understand that the UniCA setting for Time-MMD differs from the VoT setting, so the numbers are not directly comparable, but without a comparison against a covariate-adapted TSFM under the Table 2 protocol, it is hard to tell whether the gains in Table 2 hold.

Minor Weaknesses

W3. It is unclear how the training setup was configured. Please provide the hyperparameter tuning space.

Questions for authors

See weakness above.

Reviewer_1xUVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WorldTS addresses multimodal covariate-aware time series forecasting through explicit latent future-state prediction. A shared causal local-patch encoder maps historical and ground-truth future target observations into a common state space, while modality-specific modules encode numerical, textual, or visual covariates. In Stage 1, the encoders and predictor are jointly trained using latent L1 matching, cosine similarity, and VICReg regularization. In Stage 2, these modules are frozen, and a decoder is trained using both predicted states and states encoded from ground-truth future observations. The evaluation covers 12 numerical-covariate datasets, eight Time-MMD datasets, and the MMSP solar-power dataset. The paper reports competitive forecasting accuracy and ablations of the losses, local mappings, training strategy, and covariates. I view its main contribution as adapting predictive-state learning to multimodal covariate-conditioned forecasting with a prediction-first, decoding-second training procedure.

Critical strengths
  1. WorldTS achieves competitive results on 21 real-world datasets covering numerical, textual, and visual covariates. The consistent improvements over existing baselines demonstrate its effectiveness across different forecasting tasks.
  2. The paper introduces a framework that incorporates multimodal covariates into future latent-state prediction through a two-stage training strategy. This provides a useful extension of existing latent-space forecasting methods to multimodal settings.
Critical weaknesses
  1. The paper lacks direct experimental comparisons with closely related methods such as LatentTSF and TimeAlign. Therefore, it remains unclear whether WorldTS provides meaningful improvements over existing latent-space forecasting approaches.
  2. The current ablation studies do not adequately isolate the contribution of the two-stage training strategy. The one-stage baseline also removes latent-space supervision, making it unclear whether the performance gains come from the two-stage training procedure or the latent-space learning objective.
Questions for authors
  1. For each dataset and modality, what information is available at the forecast origin? Please distinguish issued forecasts from realized future measurements, describe publication-time filtering for text, and confirm that the principal baselines receive the same information.
  2. How does WorldTS compare with a simple covariate-adapted LatentTSF implementation under matched inputs, a common predictor where feasible, and comparable tuning budgets? Can this comparison separate the proposed training strategy from adding covariates and local context? 3.On MMSP, how do target-only, target plus numerical covariate, and target plus numerical and image covariate variants compare under the same backbone and training procedure? Does imagery provide a stable benefit beyond the numerical inputs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.