- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper argues that probabilistic temporal JEPAs, in particular the predictive information bottleneck variant of VJEPA (PIB-VJEPA), share the computational structure of a hidden Markov model. They introduce Markov-Chain JEPA (MCJEPA), which replaces the latent predictor with a learned transition matrix and they sketch extensions to continuous-state and continuous-time settings.
Critical strengths
- The paper offers a conceptual framework connecting probabilistic JEPA models with HMMs and relating predictive information bottlenecks to the construction of sufficient Markov representations.
Critical weaknesses
-
There are several formatting errors throughout, most notably missing citations in the Background section.
-
Several of the presented results are trivial or not novel. For example, Proposition 1 follows directly from associativity and the definition of matrix powers.
-
Theorem 1 assumes a Markov transition, a valid emission, consistent marginals, and an encoder equal to the Bayes filter. These hypotheses are essentially the definition of an HMM, so the core statement of the paper is unsurprising / tautological.
-
Experiment 4 demonstrates approximate filtering agreement in a restricted synthetic setting, but it’s unclear under what conditions you would find this agreement in general. All experiments use tiny synthetic processes, and nothing involves an actual JEPA trained on images or video.
-
Experiment 2 only compares local evidence vs. a filtering posterior using known HMM dynamics, without evaluating a learned JEPA encoder. While the authors provide conceptual connections to history-dependent JEPA encoders, a more explicit connection would strengthen this claim.
-
Experiment 4 changes both the training objective and the optimization protocol, so the comparison does not isolate the effect of changing the objective. The authors at least acknowledge this point, but it does not change my concern.
Questions for authors
I would like to better understand the motivation and significance of behind this paper. Who is the target audience? What results in this paper do you think are surprising to that target audience? What new directions of research does this paper unlock?
Thank you for your time and constructive review.
Motivation, audience, and core contributions. The paper targets researchers developing probabilistic JEPA, predictive representations, and latent world models. We agree that Proposition 1 follows from matrix powers and that Theorem 1 uses standard HMM conditions. Our contribution is not these identities themselves, but their methodological consequences for probabilistic JEPA:
(a) MCJEPA and temporal path composition (Secs. 3–4). Instead of independent horizon-specific predictors, MCJEPA learns a shared transition A, producing qtAh for every horizon. This enforces exact direct-versus-composed prediction consistency by construction (Eqs. 4–6). In Exp. 1, path disagreement is zero for MCJEPA but 0.2138, 0.1321, and 0.0645 for independent horizon predictors (Fig. 2). Under ambiguous emissions, transition error is also lower (0.0901 vs. 0.2410; Table 1), although predictive NLL is not uniformly better.
(b) Minimality versus sufficiency (Sec. 6). A Markov predictor does not imply a sufficient Markov representation. We distinguish predictive compression from residual-history testing. In Exp. 3, exhaustive enumeration of all 4,140 deterministic partitions identifies the known minimal four-state predictive representation, while residual-history gain detects the insufficient two-state representation (Δhist=0.1139; Table 2).
(c) Four levels of JEPA–HMM correspondence (Sec. 5). We distinguish architectural analogy, emission completion, sequence-level equivalence, and objective equivalence. Importantly, neither an implicit emission nor ordinary JEPA training guarantees coherent HMM sequence modeling. Theorem 1 specifies the stronger requirements, motivating the HMM-style training objectives evaluated in Exp. 4.
These contributions provide concrete design and diagnostic tools rather than claiming new general probabilistic identities.
Experiment 2 and learned filtering. We agree that Exp. 2 uses oracle posteriors from known HMM dynamics, not a learned JEPA encoder. This intentionally isolates the value of history without optimization confounds: under strongest emission ambiguity, filtering improves state accuracy from 0.6730 to 0.8433 (Fig. 3). Learned context encoders are instead evaluated in Exps. 1 and 4. In Exp. 4, model-relative filtering KL is 0.02546 (JEPA-only), 0.00436 (hybrid), and 0.00006 (HMM+filter; Table 3). These results illustrate the effects of explicit filtering supervision; standard JEPA training is not claimed to recover the exact Bayesian filter.
Experiment 4 and optimization confounds. We agree that objective and optimization differences are not fully isolated. Although the three regimes share the latent-state family and encoder architecture, JEPA-only uses minibatch training, whereas HMM+filter and hybrid use full-data updates (Appendix J.6). Moreover, sequence likelihood and filter distillation are not separately ablated. The reported sequence NLLs are 1.2753 (JEPA-only), 1.2660 (hybrid), and 1.2659 (HMM+filter). Thus, HMM+filter performs best without the JEPA loss. We present this experiment as an objective-behavior diagnostic, not causal evidence of hybrid superiority.
Significance and future research directions. The central lesson for JEPA research is that consistent multi-horizon prediction, sufficient latent representations, and Bayesian filtering are distinct properties. The proposed framework provides a shared-transition design that guarantees the first, a diagnostic for violations of the second, and explicit training objectives addressing the third. Section 4 extends the mathematical construction to continuous-state neural transitions, including settings relevant to video and partially observed control. However, all submitted experiments are synthetic; practical benefits on image/video JEPA and downstream planning remain to be established.
Formatting and missing citations. We acknowledge the unresolved citations in Section 2, particularly line 117. We will correct these using appropriate references to deep sequential latent-variable models (Krishnan et al., 2017; Gregor et al., 2018) and latent world models (Hafner et al., 2019), and streamline the exposition to distinguish established theory from our specific contributions.