acceptodds
Under review as a conference paper at ICLR 2027

Semi-Supervised Learning and Output Prediction with Mode-Dependent Dynamical Variational Autoencoders

Abstract

Predicting multidimensional time series with hidden switching dynamics is central to safety-critical decision making and data augmentation. However, standard deep generative models rarely represent operating modes explicitly. This paper augments Dynamical Variational Autoencoders (DVAEs) with a sequence of discrete latent variables governing the switching prior dynamics of the continuous latent state. The resulting Mode-Dependent DVAEs (MD-DVAEs) can be trained in a semi-supervised manner. Namely, a labeled data subset anchors discrete latent values to known modes of operation and resolves the permutation ambiguity intrinsic to unsupervised discrete latents. A novel evidence lower bound is derived that includes a term on the categorical mode encoder for supervision. Once trained, MD-DVAEs directly predict future output sequences under either a known or stochastically sampled future mode context. The approach is validated on synthetic datasets, including a comparison with switching-dynamics models and unsupervised baselines.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
dynamical variational autoencoders, Switching state-space models, Discrete latent variable models, Time-series generation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ymLWOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates semi-supervised learning for switching dynamical systems. It proposes a mode-dependent dynamical variational autoencoder (DVAE), which is essentially a sequential VAE with an additional discrete mode variable that modulates the latent state evolution. The semi-supervised learning algorithm is based on variational inference and aims to identify the dynamical modes. The method is validated on synthetic datasets.

Critical strengths

None

Critical weaknesses

Main weaknesses. The paper studies switching dynamical systems and presents its key contribution as a semi-supervised learning strategy that identifies discrete latent values.

  • This is not a novel contribution, as related works [Blau+24, Teng+20, Fassmeyer+22] have already studied semi-supervised learning for switching dynamical systems (or a similar class of models: sequential generative models with discrete modes). The paper fails to cite these critical works and, therefore, fails to explain how its contribution differs from theirs.
  • The experimental evaluation falls below the standards expected of an ICLR submission: it includes neither baselines from related works (e.g., Teng+20, Fassmeyer+22, Blau+24, Dong+20, Ansari+21) nor experiments on real-world data. Adding these baselines and experiments would require a major revision and another round of review.

Other issues

  • The assumption of independent modes is not well justified.
  • The paper characterizes switching state-space models as follows: "While switching state-space models define a joint distribution over states and modes, their unsupervised modes cannot be steered, so they cannot predict outputs under a chosen regime". This is incorrect. Since these models define a joint distribution over states and modes, one can simply fix the mode variable to a predefined value. What requires additional effort is matching mode values to their real-world interpretations, but the models can still make predictions under a chosen regime.

References

  • [Teng+20]: Semi-supervised sequential generative models. Michael Teng, Tuan Anh Le, Adam Scibior, Frank Wood.
  • [Fassmeyer+22]: Semi-Supervised Generative Models for Multi-Agent Trajectories. Dennis Fassmeyer, Pascal Fassmeyer, Ulf Brefeld.
  • [Blau+24]: A study of animal action segmentation algorithms across supervised, unsupervised, and semi-supervised learning paradigms. Ari Blau, Evan S Schaffer, Neeli Mishra, Nathaniel J Miska, The International Brain Laboratory, Liam Paninski, Matthew R Whiteway.
Questions for authors

None

Reviewer_xK3vOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper extends dynamical VAEs with discrete operating modes and uses partial mode labels to guide training. The model generates trajectories under a supplied mode schedule. Experiments evaluate reconstruction, classification, and generation on two synthetic datasets.

Critical strengths

Explicit mode supervision provides a clear way to associate latent regimes with known operating conditions and control trajectory generation.

Critical weaknesses
  1. The experiments do not adequately test the main claims. Evaluation uses synthetic systems with one supervision rate, without measuring the benefit of unlabeled data. Reconstruction and classification use a small held-out set, and variation across training seeds is not reported. Full-sequence reconstruction and reference-schedule generation also do not establish forecasting performance from an observed history.

  2. The comparisons leave the method’s advantage unclear. Reconstruction is worse than the state-space baselines, and generation results are mixed (Tables 1–2). Missing comparisons include a DVAE without modes, a mode-conditioned neural model trained on the labeled subset, and a deep switching model such as SNLDS. Some baselines also infer or sample modes while MD-DVAE receives the true schedule.

Questions for authors
  1. Does adding unlabeled data improve over training the same model on the labeled subset alone, and how does this benefit change with the supervision rate?

  2. For Table 1, are true mode labels used in constructing the latent states supplied to the mode classifier?

  3. How are realistic future mode schedules generated when they are unknown? The independent mode prior does not capture the persistent regimes in either benchmark, and reference-schedule generation does not test this limitation.

Minor: The standard VAE background and lengthy ELBO expressions obscure the contribution and could be shortened or moved to the appendix.

Reviewer_3cBxOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary
  • The main problem being identified in this paper is that "standard deep generative models rarely represent operating modes explicitly" and that this is an important problem to be solved because existing methods for modelling Switching Dynamical Systems lack the ability to synthesize future trajectories or augment datasets.

  • More concretely, the authors argue that lacking an explicit representation of the switching states in the modelling setup (including knowledge about the 'labels'/ of each mode or switching state) is the missing piece that would enable the model to synthesize future trajectories / generate new data.

  • To solve this problem, they propose a method called MD-DVAE that features two ways in which they extend DVAEs (Dynamical Variational Autoencoders):

    • A parametrization that features a dynamics model (transition model) which takes as additional input a mode (switching state) and parametrizes the latent dynamics conditional on the mode.
    • A semi-supervised training regime in which they assume their dataset to be partially labeled with the modes that govern the latent dynamics. Their proposed loss combines a self-supervised loss and supervised loss
Critical strengths

For the most part, the paper is easy to understand, and the notation and derivations are easy to follow.

Developing methods for robustly identifying (latent) switching dynamics is an important (and hard) problem to solve.

Critical weaknesses
  • Motivation / Problem Statement / Relevance

    • The main claims about the problem being addressed are not supported
    • Claim A: "standard deep generative models rarely represent operating modes explicitly"
      • This overlooks a good amount of models out there that allow us to "represent operating modes explicitly" while also following a similar latent dynamics modelling approach: ∆-SNICA (Hälvä et al., NeurIPS 2021; with per-component switching), iSDS (Balsells-Rodas et al., ICML 2024).
      • Balsells-Rodas et al.'s iSDS features the same mode-conditioned latent transition structure as the proposed MD-DVAE (with a Markov instead of an i.i.d. mode prior), and its identifiability result holds independently of the choice of mode prior, so it also covers MD-DVAE's i.i.d. prior.
      • Furthermore, Dong et al. (2020), which the paper cites, has an explicit mode-conditioned transition p(z_t | z_{t-1}, s_t) in its generative model. It marginalizes the discrete state only during inference, so, contrary to the paper's description, it can be rolled out under a chosen mode sequence.
    • Claim B: Existing switching state space models can't be used to generate conditional on a specific mode.
      • L79-83: "A structural obstacle hinders these methods: an SDS is identifiable only up to a permutation of modes, which Balsells-Rodas et al. (2024) formalize as a label-switching ambiguity. Unsupervised training therefore cannot fix mode semantics.
        While switching state-space models define a joint distribution over states and modes, their unsupervised modes cannot be steered, so they cannot predict outputs under a chosen regime."
      • At best this is only partially correct. Yes, there is label-switching ambiguity. But up to that label switching, of course any of these existing models that feature a switching model (any conditioning on a discrete state) can be used to predict outputs under any arbitrary regime/state.
      • So it only boils down to figuring out the switching ambiguity. And given the fact that these switching models (Hälvä et al. (2021), Balsells-Rodas et al. (2024)) are proven to be identifiable up to permutation invariance, then figuring out the "correct" label assignment can be achieved with simple matching algorithms (e.g., Hungarian matching) and the labels already assumed to be available.
    • Claim C: The ELBO term is "novel"
      • Given the chosen factorization, the loss presented in this paper is a straightforward combination of the generic DVAE framework ELBO loss (Girin et al 2021) with the Semi-Supervised "M2" style loss from Kingma et al. 2014 (with Gumbel-softmax sampling in place of M2's exact marginalization of the discrete variable).
  • The performance of the method is unconvincing

    • The method presented in this paper is tested against only two very simple benchmark datasets.
    • Neither of these two datasets even exhibits any nonlinearity besides the mode switching itself. So none of the nonlinear components of MD-DVAE are even tested (except switching).
    • The most basic baselines are missing: The NASCAR dataset was introduces to benchmark the rSLDS model (Linderman et al. 2017) but isn't used as a baseline method in the paper (SSM C is only non-recurrent SLDS). Neither is iSDS or ∆-SNICA included.
    • More importantly: One of the biggest claims of this paper is that some form of "semi supervised" learning is required. At a minimum this would require a baseline that actually demonstrate the need for this. For example: train MD-DVAE (without label supervision), or SNICA, or iSDS, or rSLDS and only do a post-hoc label matching. Then measure if using labels during training and not just post-hoc results in an improved performance. Even better: Do a full ablation / set of experiments on the amount of supervision during training.
  • Reproducibility of the method and results / The paper is lacking important details about the method. In it's current state it would be impossible to reproduce the results presented in this paper. To the point, where it isn't even stated whether MD-DVAE actually has a nonlinear encoder/decoder or dynamics model. So far I've only been assuming MD-DVAE is a model that has nonlinearities and isn't purely linear because it claims that it fills a gap that exists in the "standard deep generative models". Which parts of the model are neural networks? Which aprts are not? What are the network architectures? How is the model trained? What optimizer is used? Learning rates? LR Schedulers? For the gumbel softmax: are you using hard sampling or soft sampling? All of these kinds of implementation details must be included. At least somewhere in the appendix. Alternatively, publish your codebase (anonymously during review or later publicly).

Questions for authors

See weaknesses.

REFERENCES:

  1. Kingma, D. P., Rezende, D. J., Mohamed, S. & Welling, M. Semi-supervised Learning with Deep Generative Models. in Advances in Neural Information Processing Systems vol. 27 (Curran Associates, Inc., 2014).
  2. Balsells-Rodas, C., Wang, Y. & Li, Y. On the Identifiability of Switching Dynamical Systems. Preprint at https://doi.org/10.48550/arXiv.2305.15925 (2024).
  3. Hälvä, H. et al. Disentangling Identifiable Features from Noisy Data with Structured Nonlinear ICA. Preprint at https://doi.org/10.48550/arXiv.2106.09620 (2021).
  4. Linderman, S. et al. Bayesian Learning and Inference in Recurrent Switching Linear Dynamical Systems. in Proceedings of the 20th International Conference on Artificial Intelligence and Statistics 914–922 (PMLR, 2017).

Comments on the paper

1 comment

Dear Reviewers, The authors have now submitted their responses to the reviews. There is only a limited amount of time left for you to review their replies and provide any further comments.

Your feedback is crucial to ensure that the review process proceeds smoothly. We would greatly appreciate it if you could share your reactions before the discussion period closes.

Thank you very much for your work. Best regards.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.