acceptodds
Under review as a conference paper at ICLR 2027

Mixed-Effects Machine Learning: Integrating Random-Effects into Representation Learning for Longitudinal Multimodal Data

Abstract

Longitudinal multimodal time-series data present two distinct modeling challenges: (1) learning predictive representations from high-dimensional time series within individual assessments and (2) accounting for participant-specific dependence across repeated assessments. We introduce Mixed-Effects Representation Learning (MERL), which builds on the architecture-agnostic mixedML framework by combining a multimodal CNN–Transformer with a linear mixed-effects model. This formulation separates population-level representation learning from structured participant-specific effects while allowing both components to contribute to prediction. We evaluate MERL using a longitudinal simulation of multimodal physiological signals with known population-level and participant-specific structure, considering both equal and heterogeneous random-effect variances. When both population-level and participant-specific effects contribute to the outcome, MERL achieves the lowest prediction error under both variance settings, with RMSEs of and , compared with and for the same neural network architecture with one-hot participant identifiers and and without participant information. MERL remains competitive when either component is removed and estimates negligible random-effect variance when random effects are absent. Under heterogeneous random-effect variances, its estimates also reflect the relative magnitudes of the true variance components. These results demonstrate the potential of combining deep representation learning for multi- modal time series with mixed-effects modeling to exploit repeated-measures structure in longitudinal prediction.

Primary area:
learning on time series and dynamical systems
Keywords:
multimodal learning, longitudinal learning, time-series representation learning, mixed-effects learning, random-effects, repeated measures, neural networks, CNN-Transformer, participant-specific modeling, future prediction

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_pbK4OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MERL for longitudinal prediction from multimodal physiological signals. MERL builds on mixedML [1]. A CNN-Transformer learns the population-level function from biosignals and scalar covariates. A linear mixed model with a random intercept and three random slopes captures participant-specific deviations. The two parts are fit by alternating estimation. On simulated data, MERL achieves lower RMSE than the same network with or without one-hot participant identifiers when both fixed and random effects are present. MERL also recovers the ordering of the true variance components.

[1] Kilian et al., Mixed effects in machine learning: A flexible mixedML framework to add random effects to supervised machine learning regression, TMLR 2023.

Critical strengths
  1. The paper clearly separates within-assessment temporal structure from across-assessment participant variation.

  2. The simulation provides ground-truth variance components, so the authors can check the random-effects estimates.

  3. The fixed-only and random-only outcomes serve as useful sanity checks.

Critical weaknesses
  1. Limited novelty. mixedML [1] is already architecture-agnostic. MERL applies it with a CNN-Transformer and does not add a new estimation method or theory.

  2. The simulation matches the model. MERL uses the same random-effects design that generates the data. The population function is a linear sum of features. The paper justifies only the feature correlations, not the variances, noise level, or sample size. It also does not test misspecification.

  3. Missing baselines. NN and NNOHE\text{NN}_{\text{OHE}} are ablations, not competing methods. The authors cite MeNets [2], LMMNN [3, 4], DeepGLMM [5], and Mandel et al. [6], which I appreciate. MeNets, LMMNN, and Mandel et al. apply directly to this setting with the same encoder. The paper notes that these methods couple the network and random effects differently. This difference is a reason to compare against them.

  4. The biosignal part is not personalized. Only x4x_4, x5x_5, and x6x_6 have participant-specific slopes. The biosignals enter only the shared network. MERL therefore cannot model participants who respond differently to signal-derived quantities such as heart rate.

[1] Kilian et al., Mixed effects in machine learning: A flexible mixedML framework to add random effects to supervised machine learning regression, TMLR 2023.

[2] Xiong et al., Mixed effects neural networks (MeNets) with applications to gaze estimation, CVPR 2019.

[3] Simchoni and Rosset, Using random effects to account for high-cardinality categorical features and repeated measures in deep neural networks, NeurIPS 2021.

[4] Simchoni and Rosset, Integrating random effects in deep neural networks, JMLR 2023.

[5] Tran et al., Bayesian deep net GLM and GLMM, Journal of Computational and Graphical Statistics 2020.

[6] Mandel et al., Neural networks for clustered and longitudinal data using mixed effects models, Biometrics 2023.

Questions for authors
  1. Table 1 seems inconsistent with Equations 14 to 16. The random effects have mean zero and are independent of the features, so Var(ymixed)≈Var(yfixed)+Var(g)\mathrm{Var}(y^{\text{mixed}}) \approx \mathrm{Var}(y^{\text{fixed}}) + \mathrm{Var}(g). Under equal variances, the global-mean RMSEs give Var(yfixed)≈90.3\mathrm{Var}(y^{\text{fixed}}) \approx 90.3 and Var(g)≈1.9\mathrm{Var}(g) \approx 1.9. The mixed global-mean RMSE should then be near 9.6, but Table 1 reports 11.210. In the heterogeneous setting, the implied Var(g)\mathrm{Var}(g) rises from about 1.9 to about 43, although no variance increases by more than a factor of four. Can the authors report the empirical variances of ff and gg and explain this?

  2. Why are the variance components underestimated in Table 2? For example, the residual variance in the "Mixed" column is estimated at 0.792 and 0.817 under the equal and heterogeneous settings, while the true value is 1.

  3. How should practitioners choose the random-effects design on real data? How sensitive is MERL to this choice?

  4. MERL models participant-specific effects with a linear mixed model. Could the framework replace it with a more expressive nonlinear model, such as a shallow neural network over participant identifiers and participant-level features?

Reviewer_uPfSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes MERL, a method for longitudinal multimodal data prediction that integrates a CNN–Transformer architecture with a linear mixed-effects model. The CNN–Transformer component learns common, population-level patterns from multimodal physiological signals, while the mixed-effects model accounts for individual participant differences through random intercepts and random slopes; the two components are trained alternately to perform predictions. The authors conducted experiments on fully synthesized longitudinal physiological data.

Critical strengths

None

Critical weaknesses
  1. The method has limited innovativeness and lacks sufficient comparison with existing related methods. MERL is mainly built on the existing mixedML framework, implementing the Fixed-Effects Learner through CNN-Transformer, but it does not propose an estimation or optimization mechanism that is significantly different from the original mixedML. Furthermore, although the authors discussed existing neural hybrid effects methods such as MeNets and LMMNN in Related Work, they did not compare them with these methods in the experiments.

  2. The experiment relied entirely on synthetic data, and the data generation method was highly consistent with the model assumptions. All experiments in this paper used artificially synthesized data and were not validated by real longitudinal physiological data. More importantly, the random effects in the simulation data consist of a random intercept and three random slopes, and MERL happens to use the same random effects structure, with even the variables corresponding to the random slopes being exactly the same. While this setup is suitable for verifying whether a model can recover a known statistical structure, it may also bias experimental results toward the proposed method. Currently, there is a lack of real-world data experiments and assessments when the random effects structure does not conform to the model assumptions. Therefore, it is unclear whether the advantages of MERL can be extended to more complex real-world scenarios.

Questions for authors
  1. Compared to existing mixedML frameworks, what are the specific methodological innovations of MERL? MixedML already supports combining machine learning models with random effects models through alternating estimation. MERL appears to primarily replace the Fixed-Effects Learner with a CNN-Transformer.

  2. Why wasn't a direct comparison made with existing neural mixed-effects methods? The paper discusses related methods such as MeNets and LMMNN, but the experiments mainly compare ordinary NNs and NNOHE. Could the author provide a comparison with existing Neural Mixed-Effects Models? In addition, considering that simulated physiological signals are mainly controlled by parameters such as heart rate, respiratory rate and EMG Burst Duration, could the author add a baseline that extracts these features from the signal and then uses the traditional Mixed-Effects Model for prediction to verify the necessity of complex CNN-Transformer?

  3. Can the authors provide experimental results based on real-world data? All current experiments are based on synthetic data, and the simulated data primarily utilize a fixed seed for data generation. Can the authors conduct experiments using real longitudinal data?

  4. How much additional training computational overhead does MERL introduce?

Reviewer_pUhuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces Mixed-Effect Representation Learning (MERL) to separate population-level representation learning captured with multimodal CNN transformer from structured participant-specific effects. The evaluation is conducted on simulated data, where MERL achieves the lowest prediction error when both types of effects contribute to the outcome.

Critical strengths

The dissection of population-level predictive relationship from random effects is an important problem and will be of interest to the community. The proposed method is built on top of mixedML and is easy to follow. The conducted experiments on simulated data provide a useful proof of concept for the proposed method.

Critical weaknesses

Originality and significance. The paper builds on top of mixedML, where it implements the fixed-effects function with a multimodal CNN transformer but the decomposition by itself is not novel and the current set of contributions do not appear to suffice to recommend acceptance in the the version of the paper. The significance is further impacted by the limited evaluation on the synthetic data in the paper. Even in the synthetic dataset, in the heterogenous random-effect variance setting, the gains by MERL are limited.

The clarify of the paper can be further improved as well. I'd suggest to one clearly highlight the contributions and differences on top of mixedML in Section 3.3. Second, i'd suggest to separate the simulated data from the method section as it can be confusing for readers and instead position it in the experiments section. Figure 1 contains nodes that can be ignored and instead can also be used to include additional information for the understanding of the readers.

Unfortunately, in the current form I think the paper needs significant additions to be recommended for acceptance.

Questions for authors

Please refer to the weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.