Beyond Multimodal Alignment: Shared Physical Representations Across Sensors and Action Orders

Multimodal world models are often evaluated by whether different sensors produce similar representations. Yet representational similarity alone does not establish whether different sensory observations support consistent physical predictions or whether the resulting representations can be reused when familiar actions appear in new orders. We study these questions through the physical responses predicted by a model. On the Cluster Haptic dataset, audio and acceleration independently recover surface-specific behavior from disjoint observations: predictions from the two sensors are 4.5× closer for the same surface than for different surfaces on average, while both outperform a population-level prediction. We then study action-order generalization in a controlled elastoplastic system. Shared-step dynamics fit observed action programs less accurately than a whole-program predictor, but generalize substantially better to unseen action orders, with the ranking reversing on both held-out transitions across three independent initializations. Combining complementary free-decay and hysteresis observations further improves response prediction, with diagonal Gaussian beliefs giving the lowest errors among the evaluated fusion models. These results characterize cross-sensor consistency, multimodal fusion, and generalization to new action orders as distinct properties of shared physical representations.