- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to evaluate environment-code adaptation in world models when physical conditions change. The authors point out that prediction gains after an update can come either from actually learning about the changed dynamics or simply from ordinary recalibration of an imperfectly fitted initial code. To separate these two effects, they propose a matched code replay protocol that uses provably inert physical interventions as a control branch and scores both control and changed updates on the same real-change query trajectories. Using controlled experiments with CAVIA, CoDA, and a joint context model on simulated physical systems such as the rolling ball and pendulum, the paper shows that decomposing total gain into recalibration and change-specific components can change the comparison across adaptation methods and reveal harmful updates hidden by aggregate metrics.
Critical strengths
The idea of using physical laws to construct inert interventions (where nominal parameters are modified while the underlying dynamics remain unchanged, such as scaling g and L by the same factor in a pendulum) is clever and intuitive. Building on this control, the matched replay comparison cleanly separates ordinary recalibration from gains that actually come from changed observations. This decomposition is interesting and useful, as it reveals clear differences between adaptation methods on the pendulum that looking at aggregate gains alone would hide.
Critical weaknesses
My main concern is that the empirical validation is limited to simple toy settings. The primary experiments only test two 2D physical systems (a pendulum and a rolling ball) using small two-layer MLPs with simplified implementations of CAVIA and CoDA. Consequently, it is unclear whether the proposed measurement protocol changes any practical conclusions on standard benchmarks or larger world models.
Questions for authors
The paper states that the inert and unchanged branches are numerically identical and that the inert arm reproduces the unchanged arm bitwise. If so, why not directly use an unchanged branch as the control experiment, and what does the inert intervention add beyond that?