- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies time-series world models and it asks whether low forecasting error also means the model responds correctly when the action plan changes. Because models with strong MAE can still respond to interventions in the wrong direction. The paper also compares several design choices and finds that frozen latent prediction with output-side gating works well for forecasting. A directional loss is also added finally.
Critical strengths
S1. The problem is important and well motivated. Considering more about how predictions change instead of solely focusing on fitting the observed trajectory.
S2. The benchmark design is systematic.
S3. The gap between forecasting accuracy and action-response consistency is a useful empirical observation for several real applications.
Critical weaknesses
W1. I think mechanism consistency is not well measured. The metric only checks the sign of the average response to an action shift. Maybe considering the magnitude, temporal shape, dynamics would help.
W2. The directional loss is not that surprising. The training loss already enforces the same sign property used by the evaluation metric. It also does not generalize to held-out mechanisms, and some held-out relations become worse.
W3. The introduction tells more about language of world models and plan comparison. But it seems that the model is still trained only as an observational conditional forecaster instead of closed-loop or planning evaluation.
W4. The reported architectural recipe is selected and evaluated mainly on validation performance rather than an untouched test set.
W5. A direct analysis of the empirical action-outcome association is needed when explaining the clinical models.
Questions for authors
See weaknesses.