- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work proposes the cross-predictive JEPA (JEPA-x), which incorporates privileged physical states into JEPA training to constrain the model to follow physical evolution. Its learning objective consists of four source–target terms that let the encoder and predictor see representations from both observations and physical states. JEPA-x also adopts a unified privileged-state interface, which allows it to operate across different task environments without task-specific labels. Experiments show that JEPA-x achieves better prediction ability and planning performance in several simulated control environments.
Critical strengths
The unified privileged-state interface is an interesting design. It enables a single trained JEPA-x model to handle diverse manipulation scenes without task-specific labels.
Critical weaknesses
- The assumption of access to privileged physical states, even if only during training, is strong and limits the real-world applicability of JEPA-x.
- The experimental evidence is insufficient. JEPA-x is never tested on unseen environments or objects, and I do not think the current architecture can handle them. The ablations also lack the simplest variant, which only learns an encoder from Z^o_t to s_t and a predictor s{t+1} = gφ(s_t, a_t); this variant does not need SIGReg to avoid collapse. Moreover, comparing only against LeWM is not convincing: since JEPA-x has access to more information, its better performance is expected.
Questions for authors
- You designed the privileged-state interface on the encoder side so that JEPA-x can adapt to diverse environments. However, the ground-truth dynamics should also differ across environments, even with the same objects. Should the same design also be considered for the dynamics?
- Some related work (Klindt et al., 2026; Zhang et al., 2026) has shown that, under certain assumptions, JEPA can recover the physical states up to a linear transformation from observations alone, whereas your work requires the physical states to be known. How do you view this difference?