- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
MS-JEPA supplements dense future-latent prediction with interval-level summaries formed by temporal averaging. A high-level predictor estimates these summaries from observed video and feeds them to the dense predictor through cross-attention. Training first uses oracle summaries, then freezes the low-level pathway while learning summary prediction. Evaluations cover hand motion, body motion, and physical-scene forecasting.
Critical strengths
-
The construction is simple and the oracle-to-predicted comparison makes the role of guidance easy to inspect. On EgoDex, predicted summaries improve ADE/FDE/accuracy from 0.049/0.067/0.656 to 0.044/0.061/0.706.
-
Improvements extend across four datasets, several forecast horizons, and both V-JEPA2 and DINOv2 features. The appendix reports full-pipeline latency as well as accuracy, giving some context for the added computation.
Critical weaknesses
-
The central claim is not sufficiently isolated by the experiments. MS-JEPA adds a 12-layer predictor, cross-attention adapters, and staged supervision, but lacks a dense baseline matched for capacity and training compute. The sparse-state comparison is helpful, though its small advantage lacks uncertainty estimates. Attribution of the gains to interval-level modeling is the main reason for weak rejection.
-
The paper defines the abstraction loss but does not fully state the low-level and downstream loss combination, gradient paths, or Stage II duration. Results are also reported without seed variation. These omissions make the gains difficult to reproduce and assess, particularly where improvements are modest.
-
The dense predictor is trained with oracle abstractions and then frozen when predicted guidance is introduced. The oracle-versus-predicted gap does not distinguish abstraction-prediction error from sensitivity to this change in conditioning. A fine-tuning or guidance-noise control is needed to support the claim that improving the high-level predictor alone is the main remaining opportunity.
Questions for authors
-
Do the gains remain against a dense baseline matched for trainable parameters and training compute, or a dense model with an auxiliary interval-prediction loss but no guidance branch?
-
What are the exact losses, updated modules, and training durations in both stages, including the task head and any dense-latent loss? Please also specify the dataset splits used for the reported comparisons.
-
Across independent seeds, how reliable are the gains over dense prediction and sparse future states, especially the small MOVi-C and EgoExo4D differences?