PRAGMATIC DYNAMICS MODELS: LEARNING EMBODIED TEMPORAL REASONING FROM FORECASTING IN LANGUAGE

Temporal reasoning in embodied settings requires models to predict what will happen next and adapt when those predictions conflict with physical reality. Human motor control implements this through internal forward models that predict possible consequences and compare them with sensory feedback to determine whether adaptation is necessary. To model this process computationally, we introduce pragmatic dynamics models (PDMs), using language as a simulation space for this propose-then-verify planning loop. A generative model proposes possible futures, and the realized physical trajectory verifies what occurred. We then use pragmatic inference—a probabilistic model of how language distinguishes between possible world states—to identify unrealized futures that the model has difficulty distinguishing from reality. This yields a new action forecasting benchmark (PDM-Eval). The same pragmatic account then connects the verification process to preference learning via Direct Preference Optimization (DPO), where both feedback from physical reality and regularization constraints determine how to update based on the pragmatically selected comparison. Our theory also suggests an extension, which uses chain-of-thought explanations of the competing dynamics to further inform the update process. Empirically, our benchmark PDM-Eval produces more difficult comparisons than baseline benchmark creation strategies across five language models. Our proposed DPO-based adaptation algorithm also improves generalization of visual-temporal reasoning to new domains, compared to baselines tested on two vision-language backbones. Gains persist without generating explanations at test time, and only require updates to the language layers of the network, substantially reducing compute costs for fine-tuning.