- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper argues that traditional isotropic Gaussian regularization (SIGReg) in JEPA-based algorithms is not ideal because, when the learned representation is used for planning, the planner, when forced to choose between different directions, might prioritize low-variance directions over directions that arguably could take the agent closer to the goal state. The paper then introduces AnisioWM that “replaces the fixed isotropic Gaussian target with a learnable diagonal covariance under fixed- trace and anisotropy constraints”.
Critical strengths
- Overall, the paper’s key point is very clear, and I find the argument convincing.
- The theoretical results are detailed and appear correct.
- Some results, such as the visualization in Figure 3, are quite didactic.
Critical weaknesses
- Although the paper's main point seems clear, some of the jargon makes it difficult to follow some parts of the paper. An illustrative example is the sentence: “In a linear Gaussian setting, as process noise vanishes, the joint objective selects an approximately whitened representation, so Euclidean latent distance induces inverse-state- covariance weighting in state space. This places greater emphasis on low-variance directions and can change the ordering of feasible outcomes relative to the task cost, leaving positive planning regret even with exact conditional-mean prediction in representation space.”
- I struggle to understand the point of Section 2. The manuscript barely interacts with its content, and it mostly seems like a dump of citations that are barely discussed in light of the paper’s claimed contributions.
- *For the theoretical results, at the end of Proposition 1, the reduced objective is . You then set . That naturally leaves only the second term, which is SIGReg. But isn’t this result somewhat tautological? If you set one of the terms to zero, obviously the other one will dominate the solution, no? In this sense, the prediction itself is meaningless (and perfect). How realistic is this?
- *The empirical results seem quite weak. For example, in Figure 2, each number is the average over three seeds, and there’s no sense of performance spread. How meaningful is it that after all this trouble one can improve over LeWM by no more than 6% (sometimes 1%)? What is the variance in performance for LeWM and AnisioWM?
Questions for authors
- Please respond to the two weaknesses above highlighted with an asterisk.
- It is said that a regularization weight of 0.09 was used. Was this hyperparameter tuned for AnisioWM? What were the values considered? I couldn’t find this information in the Appendix, but I might have missed it as well.
- For LeWM, it is said that a regularization weight of 0.09 was used too. Why was this value selected? Was it tuned for LeWM? Was it tuned for * AnisioWM, and for consistency, was it also used in LeWM?
- For the theory, the paper considers an affine encoder. But is this what was implemented? How do I reconcile this type of encoder with the common practice of using neural networks, for example?
- On Eq. (10), what is v_i? Where is it used afterward? Was it defined?
8 comments
Response to Reviewer iR7e (2/4)
W3: The small-noise result appears tautological because prediction loss vanishes.
Role of prediction in the analysis. We understand the reviewer's concern that the result may appear to follow simply from removing the prediction term as process noise approaches zero. However, our analysis does not assume zero prediction loss. We optimize the full prediction-regularization objective at positive process noise and characterize its global minimizers as the noise decreases.
For a linear encoder , the prediction loss decomposes as
where is the conditional next-state mean and is the process-noise covariance.
The first term measures error relative to the conditional mean. The second is the irreducible prediction error, which remains positive at for a noncollapsed encoder and depends on the representation.
Importantly, Proposition 1 does not assume that the encoder is invertible or that the predictor is exact. It establishes that, for sufficiently small positive noise, every global minimizer has a noncollapsed encoder and recovers the exact encoded conditional mean.
As noise decreases, isotropic regularization determines the leading-order representation geometry. Theorem 1 then establishes the consequence for planning: the induced latent metric can select a different feasible outcome from the task cost, leaving strictly positive planning regret even as prediction loss and encoded rollout error vanish.
Thus, accurate prediction alone does not guarantee that Euclidean distances in the learned representation correctly rank candidate plans.
Finite-noise joint training. To examine whether this mismatch persists away from the small-noise limit, we additionally trained the linear encoder and predictor jointly using the full objective at four positive noise levels. The table below reports mean planning regret over ten seeds.
| Noise ordering | 0.03 | 0.01 | 0.001 | Small-noise limit | |
|---|---|---|---|---|---|
| Favorable | 0.471 | 0.498 | 0.506 | 0.512 | 0.500 |
| Reverse | 0.554 | 0.526 | 0.519 | 0.515 | 0.500 |
The trained models exhibit planning regret close to the predicted limit across the tested noise levels. At the largest noise level, the process-noise variances are 10% and 40% of the corresponding state variances. The mismatch therefore persists beyond the nearly deterministic regime.
Prediction-driven spectrum learning. Prediction also plays a direct role when the target spectrum is learnable. Proposition 2 shows that the limiting allocation minimizes , where is the state-whitened predictive-noise covariance.
The finite-noise experiments in Appendix B.3 further show that the learned allocation responds to the noise structure. Under favorable noise ordering, anisotropy reduces planning regret; reversing the noise ordering can instead increase regret. The learned spectrum is therefore influenced by prediction, while its effect on planning depends on the relationship between predictive noise and task geometry.
Response to Reviewer iR7e (3/4)
W4: The reported planning gains are small, and their variability is unclear.
A. As requested, we report the individual planning success rates and sample standard deviations for both LeWM and AnisoWM, together with paired uncertainty estimates.
Both methods are evaluated on the same 50 initial–goal pairs per task using identical evaluator and planner settings.
| Task | LeWM: 3072 / 1234 / 42 | AnisoWM: 3072 / 1234 / 42 | LeWM mean ± SD | AnisoWM mean ± SD | Mean Δ (pp) [95% CI] |
|---|---|---|---|---|---|
| TwoRoom | 84 / 86 / 88 | 90 / 94 / 94 | 86.0 ± 2.0 | 92.7 ± 2.3 | +6.7 [0.7, 14.7] |
| Reacher | 74 / 82 / 78 | 90 / 88 / 88 | 78.0 ± 4.0 | 88.7 ± 1.2 | +10.7 [0.0, 21.3] |
| PushT | 90 / 94 / 88 | 98 / 96 / 96 | 90.7 ± 3.1 | 96.7 ± 1.2 | +6.0 [0.0, 13.3] |
| Cube | 76 / 78 / 68 | 78 / 80 / 80 | 74.0 ± 5.3 | 79.3 ± 1.2 | +5.3 [−2.7, 14.7] |
We estimate uncertainty using 20,000 bootstrap draws, jointly resampling the shared evaluation pairs and independently resampling training runs within each method.
AnisoWM achieves higher mean success on all four tasks. However, when both evaluation-pair and training-run variation are considered, TwoRoom has the clearest support for improvement. The intervals for Reacher and PushT reach zero at the reported precision, while Cube's interval includes zero.
We also distinguish these matched comparisons from the published LeWM reference. As the reviewer noted, the improvement on PushT is small against the published result, whereas the matched comparison with reproduced LeWM yields a larger difference. We have clarified the sources of these comparisons in Section 5.1.
The additional results and evaluation protocol are provided in Appendix F.1.
AnisoWM also introduces no additional planning-time components. The target variances are learned during training and discarded afterward, leaving the predictor architecture and Euclidean planner unchanged.
Q2 & Q3: The choice of and its sensitivity are unclear.
Regularization-weight selection. We clarify that was adopted from the original LeWM study (Maes et al., 2026, Appendix G, Figure 16), which reports its strongest PushT performance near this value.
We did not tune specifically for AnisoWM. Both AnisoWM and reproduced LeWM use the same regularization weight in the primary comparisons.
Sensitivity analysis. To further examine the choice of regularization strength, we evaluated both methods at seven weights on PushT. The comparison uses seed 3072 and the same 50 evaluation pairs, with the remaining settings unchanged.
| 0.01 | 0.05 | 0.09 | 0.095 | 0.10 | 0.20 | 0.50 | |
|---|---|---|---|---|---|---|---|
| LeWM success (%) | 92 | 92 | 90 | 92 | 92 | 82 | 68 |
| AnisoWM success (%) | 100 | 96 | 98 | 96 | 94 | 90 | 70 |
AnisoWM achieves higher success across all tested weights, indicating that the improvement is not confined to . Both methods show reduced performance at larger regularization weights.
The sweep uses one training run per setting and is intended to characterize sensitivity rather than training-seed variability. The weight-selection rationale and comparison are reported in Appendix F.1 (Table 6).
Response to Reviewer iR7e (1/4)
We thank the reviewer for the comments on the presentation, theoretical analysis, and empirical evaluation. We have revised the explanation of the planning mismatch and related work, and conducted additional experiments on finite-noise training, nonlinear representations, and performance variability.
W1: The explanation of the planning mismatch is difficult to follow.
A. Following the reviewer's feedback, we simplified the discussion in the Introduction to explain the planning mismatch without introducing the technical terminology prematurely.
The revised passage reads:
Section 3 first introduces the state covariance and the metric induced by isotropic feature covariance (Section 3.1), then analyzes the geometry selected by joint prediction–SIGReg training under a process-noise model (Section 3.2).
W2: The Related Work section does not clearly connect prior methods to our contribution.
A. We revised Section 2 to explain how prior work relates to the planning mismatch studied in this paper. In particular, LeWM combines Euclidean latent-distance planning with SIGReg's isotropic Gaussian target. Our analysis shows that this combination can induce a planning cost that disagrees with the task cost, even with accurate prediction. Unlike approaches that modify prediction or introduce task-specific planning costs, AnisoWM addresses this mismatch by relaxing the isotropic regularization target while retaining LeWM's prediction objective and Euclidean planner.