acceptodds
Under review as a conference paper at ICLR 2027

On the Divergence of Accuracy and Mechanism Consistency in Time Series World Models

43 views

Abstract

A time series world model (TSWM) takes the observed history of a controlled system and a plan of future actions and exogenous inputs, and predicts how the state will respond. Current approaches build such models as forecasters with the actions as covariates, trained and evaluated on prediction error under the executed plan. Yet a world model is used to compare plans that were never executed, so its response to a changed plan matters as much as its error, and this practice never tests it. Two questions are therefore open: which design choices matter, and whether an accurate forecaster responds to a changed plan the way the real system does. We address both with a formalization of TSWMs and a benchmark. The formalization separates state, actions and exogenous inputs, distinguishes continuous, mode and event actions, and introduces mechanism consistency, a metric built on declared mechanisms, relations between an action and a state variable whose direction is known, such as a vasopressor raising blood pressure: it checks whether shifting an action moves the forecast in the declared direction. The benchmark consolidates eight public datasets with real actions from engineered infrastructure and clinical care, and varies the prediction space, the plan fusion and the plan encoding across seven backbones and five seeds. First, a frozen latent prediction space lowers MAE by 9.9% over observation space and gated fusion at the output by 12.7% over concatenation at the input, both on average and on all eight datasets, whereas encoding the plan over time changes the average MAE by at most 2.2%. Second, prediction error and mechanism consistency diverge: the configuration with the lowest error is at or below chance in consistency on four of the five datasets with declared mechanisms, and no design choice avoids this. Finally, directional supervision, a loss term that penalizes the wrong-signed part of the response to a shifted action, raises consistency significantly on penalized mechanisms with no change in MAE. Together they give TSWMs a recipe: a frozen latent space and output-side fusion for accuracy, and a training objective for mechanism consistency.

Primary area:
datasets and benchmarks
Keywords:
time series world models, time series forecasting, action-conditioned forecasting, benchmark, counterfactual evaluation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_qtFbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies time-series world models and it asks whether low forecasting error also means the model responds correctly when the action plan changes. Because models with strong MAE can still respond to interventions in the wrong direction. The paper also compares several design choices and finds that frozen latent prediction with output-side gating works well for forecasting. A directional loss is also added finally.

Critical strengths

S1. The problem is important and well motivated. Considering more about how predictions change instead of solely focusing on fitting the observed trajectory.

S2. The benchmark design is systematic.

S3. The gap between forecasting accuracy and action-response consistency is a useful empirical observation for several real applications.

Critical weaknesses

W1. I think mechanism consistency is not well measured. The metric only checks the sign of the average response to an action shift. Maybe considering the magnitude, temporal shape, dynamics would help.

W2. The directional loss is not that surprising. The training loss already enforces the same sign property used by the evaluation metric. It also does not generalize to held-out mechanisms, and some held-out relations become worse.

W3. The introduction tells more about language of world models and plan comparison. But it seems that the model is still trained only as an observational conditional forecaster instead of closed-loop or planning evaluation.

W4. The reported architectural recipe is selected and evaluated mainly on validation performance rather than an untouched test set.

W5. A direct analysis of the empirical action-outcome association is needed when explaining the clinical models.

Questions for authors

See weaknesses.

Reviewer_mmxqOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates an emerging research direction, namely time series world models, for forecasting future states (variables) based on historical states, plans of future actions, and exogenous inputs. The paper formulates this important task and presents a comprehensive benchmark with diversified dimensions. More importantly, it proposes a novel metric, mechanism consistency, alongside traditional precision accuracy, offering a significant insight to guide future work.

Critical strengths
  1. The time series world model is a valuable and practical research problem, as it introduces three types of actions into time series forecasting.

  2. The benchmark considers diverse datasets from different domains and examines baseline models involving key model design choices, including prediction space, plan fusion, and plan encoding.

  3. The new metric, mechanism consistency, is important for ensuring the interpretability of time series forecasting models and serves as a necessary supplement to forecast accuracy.

  4. The core idea is illustrated with real-world examples throughout the paper, which makes it easy to understand.

  5. The final recipe is beneficial for future work, although it is somewhat counterintuitive and poses significant challenges.

Critical weaknesses
  1. Some symbols could be further reviewed and chosen more carefully. For instance, "c" is sometimes used as a subscript for an action channel and sometimes denotes a state channel. At a minimum, these uses should be distinguished by font.

  2. Figure 2 is somewhat crowded, with too much text.

Questions for authors
  1. According to the last sentence of the first paragraph on the second page, are all existing datasets simulated or synthesized?

  2. What is the role of F and F^{-1} in computing the quantile shift? Is it essentially a transformation to ensure that operations are performed on [0,1]? A detailed explanation would be helpful. I understand the page limit, but since this is a core step, it would be better to present these ideas and formulae more clearly.

Reviewer_Fr18OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper argues that standard approaches to time-series forecasting do not adequately account for observable external actions that influence future outcomes. To address this limitation, the authors investigate the relationship between forecasting accuracy and mechanism consistency in time-series world models (TSWMs). They formalize the TSWM framework and evaluate design choices at three architectural levels. They also introduce a benchmark constructed from eight public datasets containing real-world actions.

Critical strengths
  1. Comprehensive empirical benchmark

The proposed benchmark adds annotations to eight datasets to support the evaluation of prediction representations, fusion strategies, and temporal plan encoders. The inclusion of four engineering datasets and four clinical datasets provides a broad empirical basis for assessing design choices in time-series world models.

  1. Well-motivated research problem supported by empirical evidence

The authors identify an important potential discrepancy between predictive accuracy and consistency with real-world mechanisms. The evaluations clearly illustrate this issue: architectures that achieve the best forecasting accuracy can nevertheless fail to capture the underlying mechanisms correctly.

Critical weaknesses
  1. Mechanism consistency alone is insufficient

Although a lack of mechanism consistency can indicate that a model has failed to capture the underlying dynamics of a task, satisfying this criterion does not guarantee that the phenomenon is modeled accurately. Because the metric checks only the direction of change, it does not establish whether the predicted magnitude of the response is accurate.

The authors introduce a small threshold to prevent models from satisfying the metric through negligible changes in the correct direction. However, this threshold does not establish that the predicted effect sizes are meaningful or accurate.

  1. Unclear utility in real-world decision-making

The work is strongly motivated by the prospect of using TSWMs to support decision-making by comparing the outcomes of hypothetical actions. However, the evaluation does not directly assess this use case, focusing instead on forecasting MAE and directional consistency.

Predicting the correct direction of change could plausibly improve action selection in a downstream decision-making system. Nevertheless, the paper provides no direct evidence that mechanism-consistent models lead to substantially better decisions.

Questions for authors

Can the authors provide an experiment demonstrating that directional supervision improves forecasting accuracy?

Could the principles underlying the proposed training objective be applied through data augmentation, for example, by directly modifying target time series to reflect increases or decreases associated with particular actions? How would models trained on such augmented datasets compare with models trained using the proposed objective?

How does directional supervision differ from existing approaches such as monotonicity regularization? How do these methods compare empirically?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.