- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors present an argument that the global collapse regularization in JEPA-based world models, such as SIGReg in LeWM, does not guarantee that the latent space keeps the relevant dynamics-aware structure for planning. They define three key dynamics relevant failure modes: Physical Invariance Collapse, Physical Distinguishability Collapse, and Counterfactual Dynamics Collapse. Each of these modes of collapses focus on different issues and can be broken down into non-impactful visual changes resulting in shifts in the learned latent space and impactful visual change or different action applied to the same state do not result in large enough shifts in the latent space.
The authors propose PhyLatent which consists of adding 5 auxiliary losses to LeWM's training procedure. These 5 losses are defined as a static visual invariance penalty, physical state grounding, future representation alignment, counterfactual action separation penalty, and latent denoising. Across Cube, Two-Room, Reacher, and PushT benchmarks, PhyLatent not only improve the overall performance but also reduces the three collapse metrics.
Critical strengths
The motivation is well presented and useful for the overall field of world modeling and the authors have done a great job and defining what sort of collapses they notice in JEPA models and what they aim to fix. Additionally, the authors provide extensive analysis over their many experiments between PhyLatent and the baseline LeWM.
Critical weaknesses
W1. Gains over the existing methods, Sub-JEPA and SMWM, are marginal, and most of the improvement over LeWM appears to come from privileged state supervision. In table 1, PhyLatent's average is within 1 point of Sub-JEPA and SMWM which could potentially just be attributed to the standard deviation (SD is between 2-4 points across all tasks). Additionally, none of the baselines as far as I can tell received the same privileged state grounding which could attribute to higher performances. In table 3, the gains over LeWM seem mostly related to PSG other than Reacher where PhyLatent is still matched by the regular LeWM baseline. The goal-separation curves also show marginal improvements over LeWM. The displacement stratification is the most compelling result, but is not reported against LeWM + PSG, so it is unclear whether PhyLatent's gains would hold against a fairer baseline that is given the same privileged information during training. Overall, while the proposed method does seem to lower the proposed collapse metrics, the actual performance doesn't seem to be impacted as much which limits the scope of the paper as even though the learned representations might not necessarily preserve the best interpretable state and action relationships, maybe it is still useful enough for task at hand.
W2. The method integrates five additional auxiliary losses that each comes with their own weights and hyperparameters and the ablations are only done on Cube where some contradictions are shown. For example, removing SVIP somehow decreases invariance collapse and removing the counterfactual dynamics roughly leaves the CF collapse metric unchanged. Removing PSG seems to be the biggest impact on success and all 3 collapse metrics which makes it seem like that the privileged training time information is what really shapes the latent space to avoid these collapses.
Questions for authors
Q1. The reported numbers across Cube, TwoRooms, Reacher, and PushT for LeWM are all lower than those reported on the official LeWM paper. From what I can tell, the average performance across the task in their paper was 85.75 which is much higher than the reported 79.92 and much closer to PhyLatent's reported average of 86.54. Is there a difference from how the benchmarks were tested in this paper and LeWM's paper or can this be attributed to just high variance? Could you explain if there were any differences?
Q2. Can you report the displacement stratification for LeWM + PSG? Since this seems to be the strongest results in the paper, the gain from 26.67 to 60, it would be interesting to see whether the PSG objective is attributing to most of the gains.
Q3. How should one interpret table 4 on what each auxiliary objective does to LeWM? Are there any other more ablations or visualization done on how the different losses shape the latent space to avoid collapse?