- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether sparse representations make action-conditioned dynamics easier to model and reveal interpretable structure in world models. It introduces LpWM, which uses RDMReg to learn nonnegative sparse representations in a JEPA world model. The theoretical results construct sparse encodings supporting approximate linear prediction and mode-factored representations of piecewise-affine systems. The empirical study compares distributed sparse representations across predictor families (AdaLN and LTI variants) and representation dimensions. In PushT, the largest improvements occur at intermediate predictor capacities, while sufficiently expressive predictors largely remove the advantage. The paper also investigates whether representation support identifies dynamical regimes.
Critical strengths
- The predictor capacity experiment is extensive and informative. In particular, LpWM is evaluated over several predictor families across five representation dimensions, with both Gaussian and VICReg baselines. The comparisons show that gains are concentrated at intermediate predictor capacities, supporting LpWM's effectiveness in that regime.
- The interpretability experiments provide useful qualitative and quantitative results about the learned code. In the Piecewise environment, sparse support follows regime boundaries even when visual cues are absent, while linear probes recover continuous position information from feature magnitudes, demonstrating a factorization of discrete state and continuous position. The manipulation experiments show that unregularized support mainly follows effector motion, while temporal regularization improves its association with object interaction.
Critical weaknesses
- The main comparison does not sufficiently isolate sparsity, and its uncertainty estimates do not establish training-level reproducibility. Several factors change simultaneously between the sparse and dense configurations: nonnegativity, target-distribution shape, and predictor output activation. The last difference directly affects the claim about predictor complexity. Table 1 uses an identity output for dense LeWM and RepReLU for sparse LpWM. Thus, the sparse “LTI” predictor has the form whereas the corresponding dense predictor has no output nonlinearity. Although the experiments establish an advantage for LpWM, they do not establish that sparsity alone accounts for it.
The error bars for PushT performance (Fig. 1b) reflect three planning-evaluation seeds rather than independent training runs (Appendix I.1).
- In Proposition 3.1, the transition matrix is an unrestricted function of the action. This differs from the additive action dependence in several experimental predictors, and the theorem itself does not bound the complexity of implementing . The approximation rate applies to fixed horizons and suffers from the curse of dimensionality. These restrictions are relatively strong and should be acknowledged when describing the result.
Questions for authors
- (Weakness 1) Do the intermediate-capacity PushT gains persist when both methods use identical predictor output architectures? Which gains remain after separately controlling nonnegativity, sparsity, and target-distribution shape?
- The mode-factorization construction explicitly uses the true mode . Is there any result linking optimization of the proposed objective to recovery of that partition?
3 comments
Thanks for addressing my comments. Training on the new seeds addresses my reproducibility concerns, and the comparison between sparse LTI and dense MLP$\circ$LTI strengthens the argument for predictor capacity. The distinction between the theoretical existence results and the empirical recovery of dynamical regimes is clearer, and Theorem 4.2 is better scoped.
Some uncertainty remains about the relative contributions of sparsity, target distribution, and training parameterization. For example, removing the predictor’s ReLU while retaining sparse encoder outputs reduces success from approximately 63% to 17%. This shows that performance depends strongly on how the predictor handles the sparse code, leaving the contribution of sparsity itself difficult to separate from that of the output link. However, given the substantial improvement in the paper, I am raising my rating from 2 to 3.
Thank you for raising your score and we're glad that we have resolved some of your concerns. Regarding "removing the predictor’s ReLU while retaining sparse encoder outputs reduces success from approximately 63% to 17%", we believe this is expected and it still isolates the benefit of sparsity.
In the JEPA architecture, the predictor is a latent dynamics model over the latent space produced by the encoder, and thus any rollout of the predictor should remain in this particular latent space instead of drifting away. Hence if the encoder produces sparse outputs (by sparse, we mean that the output has zero entries, i.e. sparsity is the outcome of the link function), then the predictor also has to output sparse predictions so that subsequent rollouts will condition on this sparse output as inputs. With ReLU only applied over encoder, this is a mismatch and the problem will actually show up earlier during training, since we minimize MSE between encoder and predictor.
As shown in our response to Reviewer hf3m, we also tried soft-threshold instead of ReLU as link function since soft-threshold is considered more as the canonical sparsity-inducing transformation in sparse coding / Iterative Shrinkage-Thresholding Algorithm (ISTA). We also observe performance improvements and hence the benefit is not coming from the particular choice of ReLU.
We hope we have clarified your concern, and let us know if you have any further questions. Your feedback has greatly strengthen our draft and we appreciate your time and efforts!
Thank you for your thoughtful and constructive feedback. We address your concerns below.
The main comparison does not sufficiently isolate sparsity
See our response to Reviewer hf3m. TLDR:
Additive action dependence is a special case of action-dependent linear dynamics.
Consider an additive (control-affine) linear predictor
zt+1=Azt+Bat.
If we append a constant coordinate, z~t=[zt⊤,,1]⊤, then
= \underbrace{\begin{bmatrix} \mathbf{A} & \mathbf{B}\mathbf{a}_t \\\\ \mathbf{0}^\top & 1 \end{bmatrix}}_{\mathbf{P}(\mathbf{a}_t)} \tilde{\mathbf{z}}_t ,$$ which is linear in the latent state with an action-dependent matrix $\mathbf{P}(\mathbf{a})$. The additive model is the special case in which $\mathbf{P}(\mathbf{a})$ depends on $\mathbf{a}$ only through its last column, affinely. We would also like to clarify that we do not intend to use Proposition 3.1 to prove theoretically that LpWM linearizes latent dynamics. Proposition 3.1 is one example that shows the existence of a solution where one-hot code linearizes dynamics. We will make sure to tone down the claim that clearly indicates the Proposition 3.1 is only used as an illustrative example. ## Others **1. (Weakness 1) Do the intermediate-capacity PushT gains persist when both methods use identical predictor output architectures? Which gains remain after separately controlling nonnegativity, sparsity, and target-distribution shape?** See our responses above. **any result linking optimization of the proposed objective to recovery of that partition** No, and we thank you for pressing on this. Theorem 4.2 takes the true mode $q$ as given: it shows that a mode-factored sparse code *exists* for piecewise-affine dynamics and that the induced latent dynamics are affine within each mode. It does not show that optimizing RDMReg recovers this partition, and we do not claim it does. We have revised the abstract and conclusion accordingly: recovery of the partition by the learned encoder is now stated as an empirical finding, not a guarantee. See our responses to Reviewer hf3m for further details. ## Conclusion We hope that we have clarified some of the concerns you have, and your feedback has been really constructive! We appreciate if you would consider raising your score, if you think we have addressed your concerns. Let us know if you have any more questions!