acceptodds
Under review as a conference paper at ICLR 2027

FORECASTING SKILL ALONE DOES NOT CERTIFY INTERVENTION COMPETENCE IN WORLD MODELS

Abstract

World models are validated by forecasting held-out trajectories, yet they are also used to predict outcomes after actions that change the system. Held-out forecast error does not directly measure whether a model can distinguish outcomes under candidate actions, although it is often treated as evidence of intervention competence. We introduce forecast-gated intervention evaluation: a world model first demonstrates held-out forecast skill and is then tested on interventions. We ask whether rolling its learned dynamics forward adds intervention value beyond label-supervised predictors that use the same observation and intervention but never roll dynamics forward. The benchmark is two simulated Epileptor cohorts (12 and 16 nodes) whose resection outcome is decided by one latent variable, a deliberately favorable case for certification. An ODE-based Epileptic-brain World Model (EWM) clears the gate but trails the no-rollout baseline in both settings: AUROC is 0.519 vs. 0.743 at 12 nodes and 0.509 vs. 0.701 at 16 nodes. Ten further rollout models pass the gate. At the primary label budget, no evaluated rollout clears the prespecified margin over the label-supervised no-rollout head built on TGR-WM's temporal encoder. Swapping the EWM and TGR-WM encoders reverses their intervention-performance ordering, suggesting that intervention transfer is strongly encoder-dependent in this benchmark. Within each EWM member, the decoded signal tracks the intervention-relevant state, but independently trained members exhibit different orientations in that association, so averaging cancels it. A validation-derived orientation bit per member brings EWM only level with the floor. World models intended for interventions should be evaluated on interventions, against strong no-rollout baselines.

Then back it, or bet against it.

Related papers

Open the market on this paper to see 7 more related papers.