Mixed-Effects Machine Learning: Integrating Random-Effects into Representation Learning for Longitudinal Multimodal Data

Longitudinal multimodal time-series data present two distinct modeling challenges: (1) learning predictive representations from high-dimensional time series within individual assessments and (2) accounting for participant-specific dependence across repeated assessments. We introduce Mixed-Effects Representation Learning (MERL), which builds on the architecture-agnostic mixedML framework by combining a multimodal CNN–Transformer with a linear mixed-effects model. This formulation separates population-level representation learning from structured participant-specific effects while allowing both components to contribute to prediction. We evaluate MERL using a longitudinal simulation of multimodal physiological signals with known population-level and participant-specific structure, considering both equal and heterogeneous random-effect variances. When both population-level and participant-specific effects contribute to the outcome, MERL achieves the lowest prediction error under both variance settings, with RMSEs of $1.525$ and $1.469$, compared with $2.470$ and $2.230$ for the same neural network architecture with one-hot participant identifiers and $6.166$ and $6.790$ without participant information. MERL remains competitive when either component is removed and estimates negligible random-effect variance when random effects are absent. Under heterogeneous random-effect variances, its estimates also reflect the relative magnitudes of the true variance components. These results demonstrate the potential of combining deep representation learning for multi- modal time series with mixed-effects modeling to exploit repeated-measures structure in longitudinal prediction.