- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ManiJEPA, a manifold-aware JEPA world model designed to improve long-horizon planning by mitigating error accumulation during autoregressive rollouts. The key motivation is that existing approaches either suffer from compounding prediction errors or impose uniform contraction on the predictor's Jacobian, which can suppress the sensitivity to early actions needed for effective planning. ManiJEPA assumes that reachable latent states lie on a low-dimensional manifold and introduces directional Jacobian regularization to distinguish between tangent and normal directions. Specifically, the method estimates the local tangent space using PCA over neighboring latent states and introduces three regularization terms: normal contraction to suppress off-manifold errors, tangent preservation to maintain control authority, and leak blocking to prevent normal errors from propagating into tangent directions. The method is evaluated on goal-conditioned planning tasks, including PushT, TwoRoom, and OGBench-Cube, as well as policy learning on LIBERO-Goal. Experimental results show improvements over JEPA-based baselines, particularly at larger goal offsets.
Critical strengths
-
The paper addresses an important challenge in latent world models: mitigating error accumulation during autoregressive rollouts while preserving the sensitivity to actions required for effective planning. The motivation is clear, and the tension between rollout stability and control authority is well articulated.
-
The proposed directional Jacobian regularization provides an interesting perspective on stabilizing latent dynamics. By distinguishing between tangent and normal directions of the latent manifold, ManiJEPA aims to suppress off-manifold errors without excessively restricting on-manifold dynamics. The theoretical analysis provides useful justification for the proposed design under the stated assumptions.
-
ManiJEPA achieves improvements over the baselines as the goal offset increases across PushT, TwoRoom, and OGBench-Cube, while maintaining competitive performance at shorter offsets. The consistent improvements across different environments provide encouraging evidence for the effectiveness of the proposed regularization.
Critical weaknesses
-
One of my main concerns is that the theoretical guarantees rely on the directional Jacobian constraints holding along the entire autoregressive rollout trajectory. However, in practice, these constraints are only applied to latent states encoded from the training data, rather than to the model's own predicted states. This is particularly important because the motivation of ManiJEPA is to address error accumulation when predictions drift away from the data manifold.
-
The proposed method assumes that reachable latent states lie on a well-defined low-dimensional manifold and that its local tangent space can be accurately estimated using PCA over neighboring latent representations. However, these assumptions may not hold in more complex environments involving discontinuous dynamics, contact-rich interactions, or insufficient state coverage. In particular, directions with low variance in the training data may still correspond to valid and important action-induced changes. Contracting such directions could introduce bias into the learned dynamics.
-
The paper emphasizes improvements in long-horizon planning, but the main experiments increase the goal offset while keeping the model-based planning horizon fixed at five latent steps (25 environment steps), with replanning between action sequences. Therefore, the reported improvements at larger goal offsets do not directly demonstrate that the model can accurately predict or plan over equally long open-loop horizons. A more systematic evaluation of long-horizon prediction accuracy and planning performance across environments would better support the central claims.
-
ManiJEPA introduces several hyperparameters that control the strength of directional regularization, including the tangent-shell bounds, normal contraction threshold, leak threshold, and corresponding loss weights. It is unclear whether these thresholds are generally applicable across different environments or require task-specific tuning.
Questions for authors
-
How sensitive is ManiJEPA to errors in the tangent-space estimation? In particular, could valid but rarely observed action-induced changes be classified as normal directions due to their low variance in the training data? How would such misclassification affect the learned dynamics and planning performance?
-
ManiJEPA improves performance on LIBERO-Goal, where the downstream policy uses an OFT action head rather than autoregressive world-model planning. What mechanism explains this improvement?
-
Could the authors clarify how the spectral norms of the Jacobian blocks, particularly those involving normal-input directions, are computed efficiently without explicitly constructing the full Jacobian? Additionally, are the local PCA computations batched across samples?
5 comments
W3: Long-Horizon Planning Claims
The reviewer notes that the main experiments use a fixed planning horizon H=5 with MPC replanning, so the offset increase reflects more replanning cycles rather than longer open-loop prediction. This is correct, and we clarify the interpretation. See Table 4.
What the offset protocol measures
The goal offset determines how many environment steps separate the initial and goal states. With H=5 latent steps (25 environment steps) and full execution before replanning, offset-25 requires one planning cycle, offset-50 requires two, offset-75 requires three, and offset-100 requires four. The protocol therefore tests whether the predictor maintains accuracy across repeated replanning cycles, which is precisely the setting where error compounding matters. If the predictor's error grew geometrically, each replanning cycle would start from a drifted latent, and the planner would fail to distinguish the goal from the current state.
Direct open-loop prediction accuracy
Figure 2(a) in the main paper directly measures open-loop prediction error: LeWM shows geometric error growth (straight line on log scale), while ManiJEPA shows at-most-linear growth (bending below the line). This is the mechanistic evidence that the predictor itself is more stable, independent of the planning protocol.
Baseline performance at longer horizons
To further decouple predictor quality from planning configuration, we re-evaluate all baselines at H=10, 15, and 20 on PushT offset-100:
| Method | H=5 | H=10 | H=15 | H=20 |
|---|---|---|---|---|
| LeWM | 12 ± 4% | 10 ± 3% | 8 ± 3% | 6 ± 2% |
| DINO-WM | 10 ± 4% | 9 ± 3% | 7 ± 3% | 5 ± 2% |
| GC-BC | 10 ± 4% | 8 ± 3% | 6 ± 2% | 4 ± 2% |
| ManiJEPA | 50 ± 6% | 58 ± 6% | 65 ± 6% | 72 ± 6% |
Baselines degrade at longer horizons because their geometric error growth compounds over more open-loop steps. ManiJEPA is the only method whose success increases with H, because its non-expansive error dynamics let the planner exploit extra lookahead without catastrophic drift. This confirms the advantage stems from predictor stability, not the fixed H=5 configuration. See Table 4.
W4: Hyperparameter Sensitivity
The paper already includes a comprehensive hyperparameter sweep (Table 10 in the appendix). Key findings: See Appendix B.11.
- Loss weights (): Performance is robust across a 100× range (0.1 to 10.0). Degradation occurs only when a weight is so small that its constraint is effectively unenforced ( approaches the w/o-normal-contraction ablation).
- Normal contraction (): The feasibility condition predicts the shape exactly, performance is stable within the feasible region () and collapses when feasibility is violated (). This is not a tuning knob but a theoretical boundary.
- Tangent shell (): The default balances error growth and authority decay. Performance degrades monotonically as increases (authority loss) or as (shell degenerates to a point).
The hyperparameters are fixed across all three control environments, so no task-specific tuning is needed within a benchmark suite. The feasibility condition provides principled guidance for setting and relative to .
Q2: LIBERO Improvement Mechanism
The reviewer asks what explains the LIBERO-Goal improvement given that the downstream policy uses an OFT action head rather than autoregressive planning. The gain reflects representation quality, not planning. See LIBERO-Goal discussion (lines 448–454) and Appendix B.6.
- The OFT action head is trained jointly with the encoder under an identical configuration for both ManiJEPA and LeWM.
- Tangent preservation keeps the on-manifold dynamics undistorted, so the action head learns from cleaner features.
- Normal contraction prevents off-manifold noise from corrupting the representation during fine-tuning, which is especially important when the policy is trained on limited data (50 demos per task).
The 16-point mean gain (71.4% → 87.4%) with no task regressing is consistent with the identifiability argument: the normal block carries no on-manifold information, so damping it should not cost performance on any individual task. This is a representation-learning benefit that transfers to any downstream head, not just CEM planning.
Q3: Efficient Spectral Norm Computation and PCA Batching
Spectral norms without dense blocks
We compute the full Jacobian using reverse-mode AD (vmap(jacrev)), which costs one backward pass per latent dimension () but is highly optimized. Given the tangent basis and normal projector , we extract blocks by projection. See revised Jacobian computation paragraph and Appendix B.3.
- , , ,
We never form or as dense matrices. Instead, we compute and via power iteration on the projected operators, which requires only matrix-vector products with (already available) and projections onto / . The cost is per iteration, not . For block (the tangent block), we compute the full SVD directly since .
PCA batching
Local PCA is batched across the full training batch (128 samples). For each sample, we find its nearest neighbors in the latent buffer and compute the local covariance matrix. The eigendecomposition is performed per-sample but vectorized across the batch using vmap. The projectors are refreshed every 100 training steps, not every step, so the amortized cost is small. The total tangent estimation overhead is included in the 1.2× wall-clock factor reported in §4.
We thank the reviewer for their careful reading and constructive feedback. Below we address each weakness and question, providing new empirical analyses where requested.
W1: Theory–Implementation Gap
The reviewer notes that Proposition 1 assumes directional bounds hold along the autoregressive rollout at z^j, while training penalties are applied at dataset latents zt. We quantify this gap empirically by rolling out the trained predictor on 2000 test initial states (PushT) and recording constraint violations at each step. See Table 3 and Appendix B.9.
At the planning horizon (25 steps), only 1.7% of rollout steps violate any constraint. Violations grow sub-linearly, consistent with bounded-error dynamics. Stratifying 500 offset-100 planning episodes by outcome:
Failures have 15× higher violation rates than successes, with normal contraction accounting for 78% of failures. The self-rollout term Lroll partially bridges this gap by training on the predictor's own outputs. We acknowledge that a fully rigorous extension would require enforcing penalties at rollout states, and we discuss this in §7 (Limitations). The empirical evidence suggests the gap is small in practice for the horizons we evaluate.
W2 & Q1: Manifold Assumption and Low-Variance Directions
The reviewer raises a subtle concern: directions with low variance in training data may still correspond to valid action-induced changes, and contracting them could introduce bias. This is the core identifiability question. See action-tangent alignment (lines 322–325), Table 2, and Appendix B.8.
Action–tangent alignment
If low-variance directions were action-relevant, we would expect action-induced displacements to point significantly into the normal space. We measure this directly: for each test latent, we compute Ja=∂Pϕ/∂a and project its range onto the estimated tangent basis, recording the smallest principal cosine:
The alignment is near-ideal (0.97–0.99) across all environments. This means action-induced displacements lie almost entirely within the tangent space, so the normal directions we contract are genuinely unobserved by the training data, not low-variance but action-relevant. The identifiability argument in §3 is therefore empirically validated: the normal block is unidentified and action-irrelevant, so contracting it introduces no bias on the dynamics that matter for planning.
Tangent vs. random subspace (non-circular validation)
To further address the circularity concern, we compare the tangent subspace against a random subspace of equal dimension in capturing actual next-step displacement Δz:
The tangent subspace captures 6–24× more displacement variance, confirming it recovers dynamically meaningful directions. See Appendix B.8.