- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
this work proposes AC-MTM, a distribution-free anti-collapse mechanism. The authors introduce a training-only inverse-dynamics head trained via an InfoNCE-style loss (Action-NCE) over in-batch action candidates. a theoretical lower bound () is provided to show that a collapsed constant encoder inevitably fails this action-identification task at chance level. Large empirical evaluations across standard continuous control suites and a harder multi-object OGBench Visual Scene task demonstrate that AC-MTM trains stably without collapse, it matches SIGReg on standard benchmarks, and significantly outperforms it on complex multi-object scenes.
Critical strengths
- an anti-collapse mechanism without distributional prior. AC-MTM replaces heuristic global priors, e.g. isotropic Gaussian matching, with a self-supervised, data-driven pressure, derived directly from sensorimotor transitions.
- theoretical justification. The information-theoretic argument, which shows that a collapsed constant encoder reduces the inverse classification problem to random guessing (), is solid.
- computational efficiency. AC-MTM preserves the same inference-time architecture, memory footprint, and CEM planning protocol as LeWM without extra cost.
Critical weaknesses
- hyperparameter dependency. The InfoNCE objective introduces sensitivity to batch composition, negative sample size, temperature , and loss weight . Although a single fixed worked across the tested suite, scaling stability to diverse environments or larger batch sizes remains unverified.
- vulnerability in weakly-controlled state dimensions (PushT & Puzzles): as the contrastive inverse loss only penalizes representations that fail to distinguish actions, state variables with weak immediate visual sensitivity, e.g. the T-block orientation in PushT or button states in visual puzzles, are underencoded, this can lead to worse performance compared to global variance priors like SIGReg.
- long-horizon generalization. The gains on OGBench Visual Scene is impressive, but they rely on a customized short-horizon trajectory-goal MPC protocol (rather than the official 750-step fixed-goal leaderboard protocol, under which both methods completely fail success rate), this raises questions about long-horizon generalization.
Questions for authors
- as Action-NCE prioritizes features that differentiate actions, how can AC-MTM be extended or modified to ensure complete coverage of task-critical static or weakly-controlled state factors such as object orientations or hidden object states without re-introducing global distribution regularizers like SIGReg?
- AC-MTM and SIGReg both fail completely () under the official 750-step fixed-goal OGBench Visual Scene protocol. Can the authors clarify what specific architectural or optimization bottlenecks prevent these one-stage pixel JEPAs from scaling to longer-horizon tasks, and how the short-horizon trajectory-goal protocol avoids these limitations?
- this method relies on in-batch action negatives to save compute. how sensitive is AC-MTM to variations in batch size ?
We'd like to thank all reviewers for their careful feedback. They led us to strengthen the comparisons and correct the paper’s positioning.
What changed in the paper: We revised the abstract, introduction and conclusion, removed the unsupported explanation that Gaussian regularization caused the Scene deficit, and added full results and protocols. We distinguish our trajectory-goal tests from the official Scene task, which both methods fail.
Why we believe the paper merits acceptance: Action-NCE provides a simple training-only alternative to SIGReg with the same planner and test-time cost. Under the original ten-epoch recipe it improves TwoRoom and Cube, roughly matches Reacher and trails PushT. Its value is a usable method together with evidence about where it helps, where it fails and how it can complement regularization.