acceptodds
Under review as a conference paper at ICLR 2027

LpWM: A Case for Sparse Representations in World Models

Abstract

Joint-embedding predictive architectures (JEPAs) learn latent dynamics for planning and avoid representation collapse by matching features to maximum-entropy distributions such as isotropic Gaussians, yielding dense representations. However, it is unclear if dense representations are the most favorable geometry for modeling dynamics. In this work, we ask whether a different geometry such as sparse representations can make action-conditioned latent dynamics easier to model, and what dynamical structure emerges from such representations. We first show that nonlinear Lipschitz dynamics can be approximated arbitrarily well by action-conditioned linear dynamics in a sufficiently high-dimensional one-hot latent space, with rollout error vanishing as the dimension grows. This motivates distributed sparse representations as a practical relaxation of one-hot sparsity. We introduce LpWorldModel (LpWM), a JEPA model regularized with Rectified Distribution Matching Regularization (RDMReg) to match encoder features to a Rectified Generalized Gaussian distribution, yielding non-negative sparse codes. Empirically, sparsity lowers the predictor complexity required for successful planning: on PushT, sparse LpWM outperforms dense LeWM by up to 57% in planning success at intermediate predictor capacities. This advantage also extends beyond Gaussian distribution matching, with LpWM outperforming dense VICReg representations across multiple predictor families. We further find and provide proven guarantee that the learned sparse representations on piecewise-affine dynamics are mode-factored, with support encoding discrete dynamical regimes and feature magnitudes capturing continuous within-regime state. Together, these results suggest that sparse representations reduce the predictor complexity for control while revealing interpretable structure.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
JEPA, Sparsity, World Models, Planning, Self-Supervised Learning, Information Maximization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_f46MOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether sparse representations make action-conditioned dynamics easier to model and reveal interpretable structure in world models. It introduces LpWM, which uses RDMReg to learn nonnegative sparse representations in a JEPA world model. The theoretical results construct sparse encodings supporting approximate linear prediction and mode-factored representations of piecewise-affine systems. The empirical study compares distributed sparse representations across predictor families (AdaLN and LTI variants) and representation dimensions. In PushT, the largest improvements occur at intermediate predictor capacities, while sufficiently expressive predictors largely remove the advantage. The paper also investigates whether representation support identifies dynamical regimes.

Critical strengths
  1. The predictor capacity experiment is extensive and informative. In particular, LpWM is evaluated over several predictor families across five representation dimensions, with both Gaussian and VICReg baselines. The comparisons show that gains are concentrated at intermediate predictor capacities, supporting LpWM's effectiveness in that regime.
  2. The interpretability experiments provide useful qualitative and quantitative results about the learned code. In the Piecewise environment, sparse support follows regime boundaries even when visual cues are absent, while linear probes recover continuous position information from feature magnitudes, demonstrating a factorization of discrete state and continuous position. The manipulation experiments show that unregularized support mainly follows effector motion, while temporal regularization improves its association with object interaction.
Critical weaknesses
  1. The main comparison does not sufficiently isolate sparsity, and its uncertainty estimates do not establish training-level reproducibility. Several factors change simultaneously between the sparse and dense configurations: nonnegativity, target-distribution shape, and predictor output activation. The last difference directly affects the claim about predictor complexity. Table 1 uses an identity output for dense LeWM and RepReLU for sparse LpWM. Thus, the sparse “LTI” predictor has the form z^t+1=ReLU(Azt+Bat),\hat{z}_{t+1} = ReLU(Az_t + Ba_t), whereas the corresponding dense predictor has no output nonlinearity. Although the experiments establish an advantage for LpWM, they do not establish that sparsity alone accounts for it.

The error bars for PushT performance (Fig. 1b) reflect three planning-evaluation seeds rather than independent training runs (Appendix I.1).

  1. In Proposition 3.1, the transition matrix Pϵ(a)P_\epsilon(a) is an unrestricted function of the action. This differs from the additive action dependence in several experimental predictors, and the theorem itself does not bound the complexity of implementing a↦Pϵ(a)a\mapsto P_\epsilon(a). The approximation rate applies to fixed horizons and suffers from the curse of dimensionality. These restrictions are relatively strong and should be acknowledged when describing the result.
Questions for authors
  1. (Weakness 1) Do the intermediate-capacity PushT gains persist when both methods use identical predictor output architectures? Which gains remain after separately controlling nonnegativity, sparsity, and target-distribution shape?
  2. The mode-factorization construction explicitly uses the true mode qq. Is there any result linking optimization of the proposed objective to recovery of that partition?
3 comments

Thank you for your thoughtful and constructive feedback. We address your concerns below.

The main comparison does not sufficiently isolate sparsity

See our response to Reviewer hf3m. TLDR:

  • (nonnegativity) We swap ReLU to soft-threshold, which allows negative entries, and the performance gain against dense still holds.
  • (target-distribution shape) We change the target-distribution shape by comparing rectified Gaussian and rectified Laplace, and we observe slight benefit when using rectified Laplace.
  • (predictor output activation) We show that we need explicit rectification on the predictor output to maintain performance. Also, if you compare LpWM with LTI(k) predictor against LeWM with MLP∘LTI(k) predictor as shown in Fig 1 (b), LpWM is still better, despite lacking an entire linear transformation.
  • (error bars) We also vary training seeds and observe no huge variations in performance.

Additive action dependence is a special case of action-dependent linear dynamics.

Consider an additive (control-affine) linear predictor

zt+1=Azt+Bat.\mathbf{z}_{t+1} = \mathbf{A}\mathbf{z}_t + \mathbf{B}\mathbf{a}_t .

If we append a constant coordinate, z~t=[zt⊤,,1]⊤\tilde{\mathbf{z}}_t = [\mathbf{z}_t^\top,\\, 1]^\top, then

= \underbrace{\begin{bmatrix} \mathbf{A} & \mathbf{B}\mathbf{a}_t \\\\ \mathbf{0}^\top & 1 \end{bmatrix}}_{\mathbf{P}(\mathbf{a}_t)} \tilde{\mathbf{z}}_t ,$$ which is linear in the latent state with an action-dependent matrix $\mathbf{P}(\mathbf{a})$. The additive model is the special case in which $\mathbf{P}(\mathbf{a})$ depends on $\mathbf{a}$ only through its last column, affinely. We would also like to clarify that we do not intend to use Proposition 3.1 to prove theoretically that LpWM linearizes latent dynamics. Proposition 3.1 is one example that shows the existence of a solution where one-hot code linearizes dynamics. We will make sure to tone down the claim that clearly indicates the Proposition 3.1 is only used as an illustrative example. ## Others **1. (Weakness 1) Do the intermediate-capacity PushT gains persist when both methods use identical predictor output architectures? Which gains remain after separately controlling nonnegativity, sparsity, and target-distribution shape?** See our responses above. **any result linking optimization of the proposed objective to recovery of that partition** No, and we thank you for pressing on this. Theorem 4.2 takes the true mode $q$ as given: it shows that a mode-factored sparse code *exists* for piecewise-affine dynamics and that the induced latent dynamics are affine within each mode. It does not show that optimizing RDMReg recovers this partition, and we do not claim it does. We have revised the abstract and conclusion accordingly: recovery of the partition by the learned encoder is now stated as an empirical finding, not a guarantee. See our responses to Reviewer hf3m for further details. ## Conclusion We hope that we have clarified some of the concerns you have, and your feedback has been really constructive! We appreciate if you would consider raising your score, if you think we have addressed your concerns. Let us know if you have any more questions!

Thanks for addressing my comments. Training on the new seeds addresses my reproducibility concerns, and the comparison between sparse LTI and dense MLP$\circ$LTI strengthens the argument for predictor capacity. The distinction between the theoretical existence results and the empirical recovery of dynamical regimes is clearer, and Theorem 4.2 is better scoped.

Some uncertainty remains about the relative contributions of sparsity, target distribution, and training parameterization. For example, removing the predictor’s ReLU while retaining sparse encoder outputs reduces success from approximately 63% to 17%. This shows that performance depends strongly on how the predictor handles the sparse code, leaving the contribution of sparsity itself difficult to separate from that of the output link. However, given the substantial improvement in the paper, I am raising my rating from 2 to 3.

Thank you for raising your score and we're glad that we have resolved some of your concerns. Regarding "removing the predictor’s ReLU while retaining sparse encoder outputs reduces success from approximately 63% to 17%", we believe this is expected and it still isolates the benefit of sparsity.

In the JEPA architecture, the predictor is a latent dynamics model over the latent space produced by the encoder, and thus any rollout of the predictor should remain in this particular latent space instead of drifting away. Hence if the encoder produces sparse outputs (by sparse, we mean that the output has zero entries, i.e. sparsity is the outcome of the link function), then the predictor also has to output sparse predictions so that subsequent rollouts will condition on this sparse output as inputs. With ReLU only applied over encoder, this is a mismatch and the problem will actually show up earlier during training, since we minimize MSE between encoder and predictor.

As shown in our response to Reviewer hf3m, we also tried soft-threshold instead of ReLU as link function since soft-threshold is considered more as the canonical sparsity-inducing transformation in sparse coding / Iterative Shrinkage-Thresholding Algorithm (ISTA). We also observe performance improvements and hence the benefit is not coming from the particular choice of ReLU.

We hope we have clarified your concern, and let us know if you have any further questions. Your feedback has greatly strengthen our draft and we appreciate your time and efforts!

Reviewer_qE2wOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates whether sparse, non-negative latent representations improve action-conditioned world models in Joint-Embedding Predictive Architectures (JEPAs). The proposed method, LpWM, regularizes a JEPA encoder using Rectified Distribution Matching Regularization (RDMReg), which matches per-timestep latent marginals to a Rectified Generalized Gaussian. Empirically, the paper evaluates planning performance across a predictor complexity ladder (from linear time-invariant to large Transformer) on three environments. It shows that the sparse LpWM outperforms dense baselines by 24–57% on PushT at intermediate predictor capacity; on the Piecewise-Affine environment, support vectors decode zone identity at near-perfect accuracy without supervision; advantages disappear at high predictor capacity. That's my read of the empirical results. There are also theoretical results. The core seems to be in Proposition 3.1 and its Corollary 3.2. The theory does suggest "sparse representations, in which a nontrivial fraction of latent coordinates are exactly zero, as a learnable relaxation that may simplify the dynamics". However, this is not an immediate conclusion, at least in my view.

Critical strengths
  1. A principled theoretical study, where the authors attempt to connect sparsity to latent linearization. There are some theoretical results that stand to suggest that sparse codes should simplify modeling.

  2. The experiments are actually very rigorous, and in my view more valuable towards arriving at the conclusions suggested by the paper.

Critical weaknesses
  1. Arguably the empirical study is somewhat narrow. The experiments are mainly on the Wall and PushT environments. They are exciting. There is an appendix that has evaluations in the PointMaze dataset as well. These are valuable. But, if this paper were only about the empirical results, then the experiment section would be limited.
Questions for authors

Q1: Do you expect the experimental results to generalize? What evidence do you suggest you have towards that end?

Q2: How do you bridge Proposition 3.1 to what you see in the experiments?

1 comment

Thank you for your thoughtful and constructive feedback. We address your concerns below.

empirical study is somewhat narrow / Q1

See our response to Reviewer hf3m for additional empirical results. Extending our methods to more challenging manipulation tasks is an ongoing efforts we are actively pursuing! Overall, I think when we move to long-horizon manipulation tasks, there are other factors that might confound with sparsity, i.e. proper subgoals, value functions etc.

bridge Proposition 3.1 to what you see in the experiments

We would also like to clarify that we do not intend to use Proposition 3.1 to prove theoretically that LpWM linearizes latent dynamics. Proposition 3.1 is one example that shows the existence of a solution where one-hot code linearizes dynamics. We will make sure to tone down the claim that clearly indicates the Proposition 3.1 is only used as an illustrative example.

Conclusion

We hope that we have clarified some of the concerns you have, and your feedback has been really constructive! We appreciate if you would consider raising your score, if you think we have addressed your concerns. Let us know if you have any more questions!

Reviewer_hf3mOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This submission presents a case study of applying Rectified LpJEPA to LeWM in a control setting. It claims that sparse latents lower the predictor capacity required for planning, and that the latent support encodes the dynamics regime, with observed improvements over dense LeWM mainly on PushT and partial evidence elsewhere.

Critical strengths

None

Critical weaknesses
  • Limited innovation. The method is Rectified LpJEPA applied to LeWM with trivial modification. The proofs are correct but concern hand-constructed encoders, not what LpWM learns, so the claimed proven guarantee on learnt representations is unsupported, meaning that the paper has no method or theory contribution and rests entirely on its empirical story.

  • Limited emperical evaluation. The sparsity-help-control claim is weakly supported. Headline gains are concentrated on single-seed PushT at intermediate predictor capacity. The LpWM linear predictors carry a ReLU output link that the LeWM ones do not, µ is fixed at 0 so sparsity is never isolated from ReLU, and no dense baseline is run on Cube. Hence, the central finding cannot be attributed to sparsity, the empirical story the paper rests on is not established.

Questions for authors
  • What in the PushT gain is attributable to sparsity rather than non-negativity or the ReLU output link? With µ fixed at 0 and no LeWM plus ReLU control, how do you rule out the latter two?

  • Which result proves the trained encoder is mode-factored? Theorem 4.2 concerns the hand-constructed encoder of Equation 7, yet the abstract claims a proven guarantee on the learned representations.

3 comments

Thank you for your thoughtful and constructive feedback. We address your concerns below.

Limited innovation

We just introduced a new regularizer as an ablation. Basically, instead of matching rectified features to rectified generalized gaussian, we can also match soft-thresholded features to soft-thresholded Laplace distribution.

The soft-threshold function is defined as Sλ(x)=sign⁡(x)⋅max⁡(∣x∣−λ,0)S_{\lambda}(x)=\operatorname{sign}(x)\cdot\max(\vert x\vert -\lambda, 0), where x∈Rx\in\mathbb{R} is the input, sign⁡(x)\operatorname{sign}(x) is -1, 0, or 1 depending on the sign of xx, and λ\lambda is the threshold value. As opposed to ReLU⁡\operatorname{ReLU}, the soft-threshold function allows negative entries.

Let our dense feature random vectors be z∈Rd\mathbf{z}\in\mathbb{R}^d. Then this regularizer is defined as Ec∼Unif⁡(Sℓ2d−1)[L(Pc⊤Sλ(z)∥Pc⊤y)]\mathbb{E}_{\mathbf{c}\sim\operatorname{Unif}(\mathbb{S}^{d-1}_{\ell_2})}[\mathcal{L}(\mathbb{P}_{\mathbf{c}^\top S_{\lambda}(\mathbf{z})}\Vert\mathbb{P}_{\mathbf{c}^\top \mathbf{y}})], where y∼∏i=1dSλ(Laplace⁡(0,σ))\mathbf{y}\sim\prod_{i=1}^{d}S_\lambda(\operatorname{Laplace}(0,\sigma)). In fact, this soft-thresholded Laplace distribution has another name. It’s also called spike-and-slab distribution with a dirac delta at zero and a Laplce slab.

See results for this regularizer in the next section.

Limited empirical evaluation

We run a few more experiments to consolidate our empirical results. For the following tables, we stick to the following setup: PushT environment, MLP∘LTI(k) predictor, feature dimension D=1536, 2 epochs of training, CEM + MPC closed loop for planning.

training seeds. In the following table, we show that “LeWM” (in fact it’s not a LeWM since we use RDMReg for Gaussian distribution matching. In one-dimensional space, Epps-Pulley are not so different from Wasserstein distance in terms of sample complexity. See [1] for more discussion) is robust to training seed variations, i.e. 11% ± 2% success rate. To save compute, we didn’t run more training seeds ablation, but there is basically no huge variations in pusht performance from varying training seed.

Training seedEval seedLeWMLpWM
000.060.68
010.120.58
020.080.56
100.100.64
110.100.68
120.160.66
200.120.62
210.120.66
220.100.58
Mean over training seeds ± std0.11 ± 0.020.63 ± 0.03

“The LpWM linear predictors carry a ReLU output link that the LeWM ones do not”. To achieve L0 sparsity, i.e. having exact zeros in our encoder output, we will have to use ReLU, or some other link function like soft-threshold. If you compare LpWM with LTI(k) predictor against LeWM with MLP∘LTI(k) predictor as shown in Fig 1 (b), LpWM is still better, despite lacking an entire linear transformation. We also tried not adding ReLU to the predictor output, and the performances drop from above 60% to 17% ± 7% for MLP∘LTI(k) predictor.

SettingClosed-loop successTraining seeds
LpWM (ReLU on encoder, no ReLU on predictor)0.17 ± 0.07 (0.24 / 0.11 / 0.15)3

“µ is fixed at 0 so sparsity is never isolated from ReLU” / What in the PushT gain is attributable to sparsity rather than non-negativity or the ReLU output link? With µ fixed at 0 and no LeWM plus ReLU control, how do you rule out the latter two?

We would like to point out that we’re using L0 norm, i.e. the number of non-zero entries in a vector, as a proxy for sparsity. We’re not using L1 norm, which is a convex surrogate for L0 sparsity.

To see if L1 sparsity also helps control, we also tried feature distribution matching towards the product Laplace distribution. [1] shows that the product Laplace distribution is the maximum-entropy distribution under expected L1 constraint (Laplace is obtained by solving the constrained optimization problem with L1 as Lagrange multiplier). In the first row of the following table, we observe that the success rate is only 13%, which is far lower than our LpWM with 63% using rectified Laplace with explicit ReLU.

We also try matching to a denser version of the rectified laplace distribution by setting μ=+1. As shown in the second row, reducing sparsity leads to 23% success rate, which is also a drop in performance.

We further compare whether different tail behavior affects performance. As shown in row 3-4, LpWM is still optimal with rectified Laplace instead of rectified Gaussian.

We also show that the soft-thresholding function leads to a different level of sparsity, i.e. 21% active entries and we also observe around 59% success rate. Since soft-threshold also allows negative entries, we show that the performance gain is not due to non-negativity. It’s also not due to ReLU since soft-threshold also works.

SettingExact zerosSignMeasured activeClosed-loop successTraining seeds
Laplace target, no ReLUnosigned1.000.131
μ=+1 (shifted rectified Laplace)fewnon-negative0.78–0.890.231
Rectified Gaussianyesnon-negative0.660.551
LpWM (rectified Laplace)yesnon-negative0.43–0.570.63 ± 0.033
Soft Threshold (threshold=0.25)yessigned0.210.691

“Which result proves the trained encoder is mode-factored? Theorem 4.2 concerns the hand-constructed encoder of Equation 7, yet the abstract claims a proven guarantee on the learned representations.”

We thank the reviewers for identifying this distinction. We agree that Lemma 4.1 and Theorem 4.2 concern the encoder explicitly constructed in Eq. (7), and do not establish that optimizing the RDMReg-regularized LpWM objective recovers this encoder or an equivalent representation. The wording in the abstract conflates the constructive guarantee with our empirical observations of learned representations. We will correct this wording consistently in the abstract, introduction, and conclusion.

The intended theoretical contribution has two parts.

First, the results proves the existential guarantee and exact dynamics of a mode-factored sparse representation that empirical results suggest. Under the assumptions of Section 4.2, Lemma 4.1 constructs a nonnegative, injective encoder with at most (r+1) active coordinates in an ambient space of dimension (m(r+1)). The active block identifies the dynamical mode, while the coefficients within that block recover the continuous state exactly. For piecewise-affine physical dynamics, Theorem 4.2 then shows that the finite-horizon latent trajectory is affine in the initial active coordinates and actions on each region realizing a fixed mode sequence. This provides a constructive mathematical account of how discrete modes and continuous state can be represented separately while preserving the system’s dynamics.

Second, this construction yields a useful simplification of planning. Corollary H.3 shows that, conditional on a fixed mode sequence, planning with a squared latent goal objective reduces to a convex quadratic program, provided the corresponding feasible action set is a nonempty polyhedron. Thus, the mode-factored representation exposes a decomposition of the planning problem into discrete mode-sequence selection and tractable continuous optimization within each sequence.

The connection to the learned LpWM representation is empirical. In Section 4.1 and Figure 2, zone identity is accurately decoded from binary support, retaining feature magnitudes improves continuous-position decoding, and support similarities reflect zone boundaries even when visual zone cues are removed. These observations provide evidence of an organization related to the theoretical construction. They do not establish that the learned encoder recovers the construction or satisfies its exact affine dynamics.

We will therefore explicitly distinguish three claims: (i) a lossless mode-factored sparse representation exists under the stated assumptions; (ii) this representation admits affine dynamics conditional on a mode sequence and the planning consequence in Corollary H.3; and (iii) trained LpWM representations empirically exhibit related support-based organization. The formal results establish the first two claims, while the experiments provide evidence for the third. Establishing when RDMReg training recovers such a representation remains an open question.

Conclusion

We hope that we have clarified some of the concerns you have, and your feedback has been really constructive! We appreciate if you would consider raising your score, if you think we have addressed your concerns. Let us know if you have any more questions!

References

[1] Rectified LpJEPA: Joint-Embedding Predictive Architectures with Sparse and Maximum-Entropy Representations. https://arxiv.org/abs/2602.01456

Comments on the paper

2 comments

Thank you for your thoughtful and constructive feedback, and especially for the careful reading of our proofs. We agree with most of your points, and we address them below.

The planning comparison confounds sparsity with RDMReg, non-negativity, RepReLU, and the choice of target distribution (C8, C9, (f), (g))

See our response to Reviewer hf3m for the full tables. TLDR:

  • (exact zeros vs. L1 sparsity) Distribution matching towards a Laplace target without ReLU only reaches 13% success rate, close to LeWM (11%). The benefit comes from exact zeros (L0 sparsity), not from the heavy-tailed L1 target.
  • (non-negativity) We swap ReLU for a soft-threshold, which allows negative entries, and still get 69% success rate, so the gain is not due to non-negativity.
  • (target-distribution shape) A rectified Gaussian reaches 55% vs. 63% for the rectified Laplace, so the target shape gives a small additional benefit on top of exact zeros.
  • (same output link, less sparsity) Keeping LpWM's RepReLU but shifting the target to μ=+1 (78–89% active entries) drops the success rate to 23%.
  • (predictor output link) Removing the ReLU from the predictor output drops LpWM to 17% ± 7%. We need explicit rectification on the predictor output, since the predictor has to output exact zeros to match the sparse targets. Also, if you compare LpWM with LTI(k) predictor against LeWM with MLP∘LTI(k) predictor as shown in Fig 1 (b), LpWM is still better, despite lacking an entire linear transformation. We will describe the LpWM rows in Table 1 as "linear core + RepReLU link" instead of linear predictors.
  • (training seeds) LpWM 63% ± 3% vs. LeWM 11% ± 2% over 3 training seeds × 3 eval seeds.

Proposition 3.1 notation and the quantizer (C1, C2, C3, S5)

Thank you for these. We will define the quantizer as the nearest representative, Qε(x)=arg⁡min⁡i∥x−ci∥Q_\varepsilon(\mathbf{x}) = \arg\min_i \lVert\mathbf{x}-\mathbf{c}_i\rVert with ties broken by the smallest index. It is deterministic, its preimage cells Qε−1(i)Q_\varepsilon^{-1}(i) partition X\mathcal{X}, and ∥x−cQε(x)∥≤ε\lVert\mathbf{x}-\mathbf{c}_{Q_\varepsilon(\mathbf{x})}\rVert\le\varepsilon still holds since the balls cover X\mathcal{X}. With distinct representatives this rule also gives Qε(ci)=iQ_\varepsilon(\mathbf{c}_i)=i, so Eε(ci)=eiE_\varepsilon(\mathbf{c}_i)=\mathbf{e}_i (the proof does not rely on it). We will state that ii is a codebook index and not a time index, write Pε(a)=[eτa(1)⋯eτa(N)]\mathbf{P}_\varepsilon(\mathbf{a}) = [\mathbf{e}_{\tau_{\mathbf{a}}(1)} \cdots \mathbf{e}_{\tau_{\mathbf{a}}(N)}] with τa(i)=Qε(f(ci,a))\tau_{\mathbf{a}}(i)=Q_\varepsilon(f(\mathbf{c}_i,\mathbf{a})), and introduce EεE_\varepsilon, ei\mathbf{e}_i, CεC_\varepsilon and QεQ_\varepsilon with one sentence each.

"Linear dynamics" is only linear in the latent state (C4, S6)

Thanks for pushing on this! We will consistently say "linear in the latent state for each fixed action". We would also like to point out that the additive action dependence used in our predictors is a special case of the action-dependent linear map in Proposition 3.1: for one-hot codes 1⊤zt=1\mathbf{1}^\top\mathbf{z}_t=1, so Azt+Bat=(A+Bat1⊤) zt=P(at) zt\mathbf{A}\mathbf{z}_t+\mathbf{B}\mathbf{a}_t = (\mathbf{A}+\mathbf{B}\mathbf{a}_t\mathbf{1}^\top)\,\mathbf{z}_t = \mathbf{P}(\mathbf{a}_t)\,\mathbf{z}_t. See our response to Reviewer f46M for the derivation. The general P(a)\mathbf{P}(\mathbf{a}) in the proposition is richer than this additive form, and we will discuss the lack of structure in the action dependence.

Proposition 3.1 is a motivating existence result (C5, S8)

We have updated our claim. We do not intend to use Proposition 3.1 to prove that LpWM linearizes latent dynamics; it is an illustrative example showing that a one-hot code which linearizes the dynamics exists. We will make this explicit, and we will state the cost: for X=[0,1]d\mathcal{X}=[0,1]^d the code needs N=(d/(2ε))dN=(\sqrt{d}/(2\varepsilon))^{d} dimensions, exponential in the state dimension dd. The learned distributed sparse code is a different object, and its connection to the one-hot construction is empirical.

The rollout limit is only for a fixed horizon (C6, (d))

We will qualify the abstract and introduction as a fixed-horizon statement. We would also like to add one refinement: for contractive dynamics (L<1L<1), the bound is uniform in the horizon, ∥xH−x^H∥≤ε∑k=0HLk≤ε/(1−L)\|\mathbf{x}_H-\hat{\mathbf{x}}_H\| \le \varepsilon\sum_{k=0}^{H}L^k \le \varepsilon/(1-L). For L=1L=1 it grows linearly in HH, and for L>1L>1 exponentially.

The mode-factored theorem is a construction, not a learning guarantee (C7, (e), S2, S7, S10)

See our response to Reviewer hf3m for details. TLDR: Lemma 4.1 and Theorem 4.2 concern the encoder constructed in Eq. (7). We will distinguish (i) a mode-factored sparse representation exists under the stated assumptions; (ii) it admits affine dynamics conditional on a mode sequence, with the planning consequence in Corollary H.3; and (iii) trained LpWM representations empirically show related support-based organization. We will remove "proven guarantee" from the abstract and conclusion, and adopt your suggested wording: "We further construct a mode-factored sparse representation for piecewise-affine dynamics and empirically observe related structure in LpWM."

Piecewise support decoding is evidence of correlation (C10, S11)

We will soften the wording to "the support decodes the discrete zone label in the Piecewise experiments" instead of claiming that support-based mode factorization is established in general. As you note, the no-visual-cue variant shows the partition is inferred from the dynamics rather than from appearance.

OGBench-Cube depends on the temporal prior (C11, (h), S12)

Without the Temporal-Jaccard regularizer, the support mostly tracks effector motion, and contact alignment requires TJ. We will state explicitly in the main text and conclusion that contact tracking does not follow from vanilla RDMReg alone, and present it as what sparse support enables under an appropriate temporal prior rather than a general consequence of sparsity.

Predictor capacity and compute (C12, S3)

We will clarify that the claimed saving is in the predictor capacity needed for successful planning (Table 2 lists predictor parameter counts at every latent dimension). The width sweep and the predictor ladder support different claims: the ladder shows where sparsity helps, and the width sweep shows that the effect holds across latent dimensions.

Planning results should be connected to dynamics prediction (C13)

We agree that planning success also depends on the planner and the goal metric. Open-loop evaluation in our figures measures a single plan without replanning, which more directly reflects the accumulated model-prediction error than closed-loop evaluation. We will make this connection explicit in the text.

Reproducibility (C14, S13)

We will consolidate the search spaces, selection rule, number of trials and seeds into a single table. The comparison is not tuned in favor of LpWM: LeWM was swept over a wider grid than LpWM (regularization weight down to 1e-4 and learning rate down to 5e-6). All methods use identical CEM budgets, candidate counts, action bounds and evaluation seeds.

Style and presentation (S1, S4, S9)

Thank you, we will fix all of these: consistent naming of LpWM / LpWorldModel and visually separating the linear core from the final link σ in Table 1.

Conclusion

We hope that we have clarified some of the concerns you have, and your feedback has been really constructive! We appreciate if you would consider raising your score, if you think we have addressed your concerns. Let us know if you have any more questions!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.