Modeling Multimodal Sequences as Heterogeneous Observations of Continuous Latent Dynamics

Multimodal sequences involve two temporal structures that are often treated as one: the evolution of task-relevant information over time and the modality-dependent grids on which that information is observed. We study a formulation in which temporally evolving multimodal evidence is represented as heterogeneous observations of a shared latent process indexed by chronological time. We instantiate this formulation with Behavioral-Time Semantic Flow (BTS-Flow), a continuous time state estimation model that propagates a latent state between observation events and revises its estimate when new multimodal evidence becomes available. Modality specific observation functions connect the shared latent state to distinct representation spaces, while a decomposition into state related and private components allows modality dependent residual information to remain outside the common dynamics. For acoustic and visual streams, timestamp based local derivatives provide an additional constraint that relates observed representation changes to the pushforward of the latent velocity through the corresponding observation functions. Language remains event valued and enters the latent state only through observation updates. Across CMU-MOSI, CMU-MOSEI, and CH-SIMS-v2, BTS-Flow improves standard predictive metrics over the strongest reported baselines. More importantly, controlled changes to observation schedules and test time temporal sparsification show lower sensitivity to the sampling grid, while held out observation inference and velocity consistency analyses support the temporal behavior implied by the formulation. These results provide empirical evidence that modeling multimodal sequences as heterogeneous observations of continuous latent dynamics is a useful inductive principle for temporally heterogeneous multimodal learning.