- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper analyzes JEPA-style world model architectures noting that high predictive accuracy in latent space does not entail representation of the features salient to decisionmaking. They view the latent space as comprising "decision" and "nuisance" components. They analyze theoretically pathologies of the dominant JEPA training objectives that cause it to latch onto nuisance components at the expense of decision components, propose a fix that amounts to encouraging invariance to nuisance dimensions, and then show that this surpasses JEPA on 2 toy control problems.
Critical strengths
None
Critical weaknesses
- The method appears to assume privileged access to histories that are the same w.r.t. decision-relevant features but different w.r.t. nuisance features. This is basically saying that someone has already labeled the solution to the problem.
- But you don't need to label the solution to the problem because it's already solved by the prior art. And there is no comparison to inverse dynamics, k-step inverses, quasimetrics, etc -- which are great ways of solving this exact problem! That whole literature is ignored. John Langford is not even cited, much less compared to. If the authors are curious they could look into k-step inverses; sensorimotor world models from Scholkopf's group is a more gentle (but much less cited) version of essentially the same idea in a JEPA setting.
- Experiments are very toy. Would be easily solved by much prior work such as AC-State, which also does much harder state representation learning.
- Theorems are trivial once you wade through the AI slop. Theorem 4.1 just says "if there's both decision-relevant feature and decision-irrelevant features then there exist representations that basically only encode one or the other, and just looking at the entropy of the resulting distribution won't tell you which". Theorem 4.2 is better illustrated with a picture or a 2D latent space example
- Writing is extremely hard to understand, at least to me, and I'm reasonably fluent in the world model literature. It reeks of claude-y faux-sophistication, and behind the veneer of technical depth is little of substance that I can dig out. It's plagued by more ancillary AI-isms, such as repeatedly defining the same notation (e.g., its notation for ReLU is defined multiple times, as though each section was made by a different chat)
Questions for authors
The only thing that could change my assessment is:
- Do your current results assume privileged access to information about the quotient set of histories under the span of nuisance variables?
- How do your results compare to just adding an inverse dynamics term to your loss?
- Can you beat strong baselines (like those under critical weaknesses) on harder benchmarks?
Dear authors and reviewers,
The author-reviewer discussion period closes on November 18 (AoE). Please engage in response periods so that we have our issues discussed thoroughly.
AC