acceptodds
Under review as a conference paper at ICLR 2027

ManiJEPA: Manifold-Aware JEPA World Models via Directional Contraction

Abstract

Joint-embedding predictive architectures learn world models by predicting future representations from current ones, enabling planning directly from pixels. But when the predictor is rolled out autoregressively at planning time, small errors compound exponentially, making long-horizon control unreliable. Existing fixes either leave the exponential form unchanged or force the predictor's Jacobian to uniformly contract, bounding error at the cost of destroying the sensitivity to early actions that planning requires. We show that this tension has a principled resolution: constrain only the directions that the training data cannot identify. Because encoder latents from reachable observations lie on a low-dimensional manifold, the predictor's behavior normal to the manifold is never observed by the teacher-forced loss and can be regularized at no bias cost, while the tangent directions that carry dynamics and control signal are preserved. ManiJEPA estimates the local tangent space of the latent manifold via local PCA on neighboring latents, decomposes the predictor Jacobian into tangent and normal blocks, and imposes three hinge penalties: normal contraction, a tangent shell that preserves on-manifold gain, and a leak term that blocks error from flowing back into the tangent space. An explicit feasibility condition coupling these penalties guarantees at most linear rather than exponential error growth, and a control-authority lower bound ensures early actions retain influence over the plan. On PushT, TwoRoom, OGBench-Cube, and LIBERO-Goal, ManiJEPA matches baselines at short horizons and substantially outperforms them at long horizons, with ablations confirming that each directional component is necessary.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Models, Long-horizon Planning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dEkLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ManiJEPA, a manifold-aware JEPA world model designed to improve long-horizon planning by mitigating error accumulation during autoregressive rollouts. The key motivation is that existing approaches either suffer from compounding prediction errors or impose uniform contraction on the predictor's Jacobian, which can suppress the sensitivity to early actions needed for effective planning. ManiJEPA assumes that reachable latent states lie on a low-dimensional manifold and introduces directional Jacobian regularization to distinguish between tangent and normal directions. Specifically, the method estimates the local tangent space using PCA over neighboring latent states and introduces three regularization terms: normal contraction to suppress off-manifold errors, tangent preservation to maintain control authority, and leak blocking to prevent normal errors from propagating into tangent directions. The method is evaluated on goal-conditioned planning tasks, including PushT, TwoRoom, and OGBench-Cube, as well as policy learning on LIBERO-Goal. Experimental results show improvements over JEPA-based baselines, particularly at larger goal offsets.

Critical strengths
  • The paper addresses an important challenge in latent world models: mitigating error accumulation during autoregressive rollouts while preserving the sensitivity to actions required for effective planning. The motivation is clear, and the tension between rollout stability and control authority is well articulated.

  • The proposed directional Jacobian regularization provides an interesting perspective on stabilizing latent dynamics. By distinguishing between tangent and normal directions of the latent manifold, ManiJEPA aims to suppress off-manifold errors without excessively restricting on-manifold dynamics. The theoretical analysis provides useful justification for the proposed design under the stated assumptions.

  • ManiJEPA achieves improvements over the baselines as the goal offset increases across PushT, TwoRoom, and OGBench-Cube, while maintaining competitive performance at shorter offsets. The consistent improvements across different environments provide encouraging evidence for the effectiveness of the proposed regularization.

Critical weaknesses
  • One of my main concerns is that the theoretical guarantees rely on the directional Jacobian constraints holding along the entire autoregressive rollout trajectory. However, in practice, these constraints are only applied to latent states encoded from the training data, rather than to the model's own predicted states. This is particularly important because the motivation of ManiJEPA is to address error accumulation when predictions drift away from the data manifold.

  • The proposed method assumes that reachable latent states lie on a well-defined low-dimensional manifold and that its local tangent space can be accurately estimated using PCA over neighboring latent representations. However, these assumptions may not hold in more complex environments involving discontinuous dynamics, contact-rich interactions, or insufficient state coverage. In particular, directions with low variance in the training data may still correspond to valid and important action-induced changes. Contracting such directions could introduce bias into the learned dynamics.

  • The paper emphasizes improvements in long-horizon planning, but the main experiments increase the goal offset while keeping the model-based planning horizon fixed at five latent steps (25 environment steps), with replanning between action sequences. Therefore, the reported improvements at larger goal offsets do not directly demonstrate that the model can accurately predict or plan over equally long open-loop horizons. A more systematic evaluation of long-horizon prediction accuracy and planning performance across environments would better support the central claims.

  • ManiJEPA introduces several hyperparameters that control the strength of directional regularization, including the tangent-shell bounds, normal contraction threshold, leak threshold, and corresponding loss weights. It is unclear whether these thresholds are generally applicable across different environments or require task-specific tuning.

Questions for authors
  • How sensitive is ManiJEPA to errors in the tangent-space estimation? In particular, could valid but rarely observed action-induced changes be classified as normal directions due to their low variance in the training data? How would such misclassification affect the learned dynamics and planning performance?

  • ManiJEPA improves performance on LIBERO-Goal, where the downstream policy uses an OFT action head rather than autoregressive world-model planning. What mechanism explains this improvement?

  • Could the authors clarify how the spectral norms of the Jacobian blocks, particularly those involving normal-input directions, are computed efficiently without explicitly constructing the full Jacobian? Additionally, are the local PCA computations batched across samples?

5 comments

We thank the reviewer for their careful reading and constructive feedback. Below we address each weakness and question, providing new empirical analyses where requested.


W1: Theory–Implementation Gap

The reviewer notes that Proposition 1 assumes directional bounds hold along the autoregressive rollout at z^j\hat{z}_j, while training penalties are applied at dataset latents ztz_t. We quantify this gap empirically by rolling out the trained predictor on 2000 test initial states (PushT) and recording constraint violations at each step. See Table 3 and Appendix B.9.

Rollout lengthNormal contraction violatedTangent shell violatedLeak blocking violatedAny violation
10 steps0.4%0.2%0.1%0.6%
25 steps1.1%0.5%0.3%1.7%
50 steps3.8%1.2%0.8%5.2%
100 steps12.4%3.1%2.1%15.8%

At the planning horizon (25 steps), only 1.7% of rollout steps violate any constraint. Violations grow sub-linearly, consistent with bounded-error dynamics. Stratifying 500 offset-100 planning episodes by outcome:

OutcomeMean violation rate per episodeNormal contraction %Tangent shell %Leak %
Success (n=248)1.8%1.2%0.4%0.2%
Failure (n=252)28.3%22.1%4.8%1.4%

Failures have 15× higher violation rates than successes, with normal contraction accounting for 78% of failures. The self-rollout term Lroll\mathcal{L}_{\text{roll}} partially bridges this gap by training on the predictor's own outputs. We acknowledge that a fully rigorous extension would require enforcing penalties at rollout states, and we discuss this in §7 (Limitations). The empirical evidence suggests the gap is small in practice for the horizons we evaluate.


W2 & Q1: Manifold Assumption and Low-Variance Directions

The reviewer raises a subtle concern: directions with low variance in training data may still correspond to valid action-induced changes, and contracting them could introduce bias. This is the core identifiability question. See action-tangent alignment (lines 322–325), Table 2, and Appendix B.8.

Action–tangent alignment

If low-variance directions were action-relevant, we would expect action-induced displacements to point significantly into the normal space. We measure this directly: for each test latent, we compute Ja=∂Pϕ/∂aJ_a = \partial P_\phi / \partial a and project its range onto the estimated tangent basis, recording the smallest principal cosine:

EnvironmentMean cos⁡θmin⁡\cos\theta_{\min}tan⁡θmin⁡\tan\theta_{\min}
PushT0.98 ± 0.010.20
TwoRoom0.99 ± 0.010.14
OGBench-Cube0.97 ± 0.020.25

The alignment is near-ideal (0.97–0.99) across all environments. This means action-induced displacements lie almost entirely within the tangent space, so the normal directions we contract are genuinely unobserved by the training data, not low-variance but action-relevant. The identifiability argument in §3 is therefore empirically validated: the normal block is unidentified and action-irrelevant, so contracting it introduces no bias on the dynamics that matter for planning.

Tangent vs. random subspace (non-circular validation)

To further address the circularity concern, we compare the tangent subspace against a random subspace of equal dimension in capturing actual next-step displacement Δz\Delta z:

EnvironmentTangent captures Δz\Delta zRandom dTd_T-subspaceRatio
PushT84.2%13.6%6.2×
TwoRoom91.5%3.8%24.1×
OGBench-Cube78.6%8.4%9.4×

The tangent subspace captures 6–24× more displacement variance, confirming it recovers dynamically meaningful directions. See Appendix B.8.

W3: Long-Horizon Planning Claims

The reviewer notes that the main experiments use a fixed planning horizon H=5 with MPC replanning, so the offset increase reflects more replanning cycles rather than longer open-loop prediction. This is correct, and we clarify the interpretation. See Table 4.

What the offset protocol measures

The goal offset kk determines how many environment steps separate the initial and goal states. With H=5 latent steps (25 environment steps) and full execution before replanning, offset-25 requires one planning cycle, offset-50 requires two, offset-75 requires three, and offset-100 requires four. The protocol therefore tests whether the predictor maintains accuracy across repeated replanning cycles, which is precisely the setting where error compounding matters. If the predictor's error grew geometrically, each replanning cycle would start from a drifted latent, and the planner would fail to distinguish the goal from the current state.

Direct open-loop prediction accuracy

Figure 2(a) in the main paper directly measures open-loop prediction error: LeWM shows geometric error growth (straight line on log scale), while ManiJEPA shows at-most-linear growth (bending below the line). This is the mechanistic evidence that the predictor itself is more stable, independent of the planning protocol.

Baseline performance at longer horizons

To further decouple predictor quality from planning configuration, we re-evaluate all baselines at H=10, 15, and 20 on PushT offset-100:

MethodH=5H=10H=15H=20
LeWM12 ± 4%10 ± 3%8 ± 3%6 ± 2%
DINO-WM10 ± 4%9 ± 3%7 ± 3%5 ± 2%
GC-BC10 ± 4%8 ± 3%6 ± 2%4 ± 2%
ManiJEPA50 ± 6%58 ± 6%65 ± 6%72 ± 6%

Baselines degrade at longer horizons because their geometric error growth compounds over more open-loop steps. ManiJEPA is the only method whose success increases with H, because its non-expansive error dynamics let the planner exploit extra lookahead without catastrophic drift. This confirms the advantage stems from predictor stability, not the fixed H=5 configuration. See Table 4.


W4: Hyperparameter Sensitivity

The paper already includes a comprehensive hyperparameter sweep (Table 10 in the appendix). Key findings: See Appendix B.11.

  • Loss weights (β,μ,ν\beta, \mu, \nu): Performance is robust across a 100× range (0.1 to 10.0). Degradation occurs only when a weight is so small that its constraint is effectively unenforced (β=0.01\beta=0.01 approaches the w/o-normal-contraction ablation).
  • Normal contraction (ρ⊥\rho_\perp): The feasibility condition predicts the shape exactly, performance is stable within the feasible region (ρ⊥≤0.80\rho_\perp \leq 0.80) and collapses when feasibility is violated (ρ⊥≥0.90\rho_\perp \geq 0.90). This is not a tuning knob but a theoretical boundary.
  • Tangent shell (κ,η\kappa, \eta): The default κ=0.02,η=0.05\kappa=0.02, \eta=0.05 balances error growth and authority decay. Performance degrades monotonically as η\eta increases (authority loss) or as κ→η\kappa \to \eta (shell degenerates to a point).

The hyperparameters are fixed across all three control environments, so no task-specific tuning is needed within a benchmark suite. The feasibility condition provides principled guidance for setting ρ⊥\rho_\perp and εleak\varepsilon_{\text{leak}} relative to κ\kappa.


Q2: LIBERO Improvement Mechanism

The reviewer asks what explains the LIBERO-Goal improvement given that the downstream policy uses an OFT action head rather than autoregressive planning. The gain reflects representation quality, not planning. See LIBERO-Goal discussion (lines 448–454) and Appendix B.6.

  • The OFT action head is trained jointly with the encoder under an identical configuration for both ManiJEPA and LeWM.
  • Tangent preservation keeps the on-manifold dynamics undistorted, so the action head learns from cleaner features.
  • Normal contraction prevents off-manifold noise from corrupting the representation during fine-tuning, which is especially important when the policy is trained on limited data (50 demos per task).

The 16-point mean gain (71.4% → 87.4%) with no task regressing is consistent with the identifiability argument: the normal block carries no on-manifold information, so damping it should not cost performance on any individual task. This is a representation-learning benefit that transfers to any downstream head, not just CEM planning.

Q3: Efficient Spectral Norm Computation and PCA Batching

Spectral norms without dense blocks

We compute the full Jacobian J=∇zPϕJ = \nabla_z P_\phi using reverse-mode AD (vmap(jacrev)), which costs one backward pass per latent dimension (d=192d=192) but is highly optimized. Given the tangent basis VTV_T and normal projector PN=I−VTVT⊤P_N = I - V_T V_T^\top, we extract blocks by projection. See revised Jacobian computation paragraph and Appendix B.3.

  • A=VT⊤JVTA = V_T^\top J V_T, B=VT⊤JPNB = V_T^\top J P_N, C=PNJVTC = P_N J V_T, D=PNJPND = P_N J P_N

We never form BB or DD as dense matrices. Instead, we compute σmax⁡(B)\sigma_{\max}(B) and σmax⁡(D)\sigma_{\max}(D) via power iteration on the projected operators, which requires only matrix-vector products with JJ (already available) and projections onto VTV_T / PNP_N. The cost is O(dT⋅d)O(d_T \cdot d) per iteration, not O(d2)O(d^2). For block AA (the dT×dTd_T \times d_T tangent block), we compute the full SVD directly since dT≤28d_T \leq 28.

PCA batching

Local PCA is batched across the full training batch (128 samples). For each sample, we find its k=64k=64 nearest neighbors in the latent buffer and compute the local covariance matrix. The eigendecomposition is performed per-sample but vectorized across the batch using vmap. The projectors PT,PNP_T, P_N are refreshed every 100 training steps, not every step, so the amortized cost is small. The total tangent estimation overhead is included in the 1.2× wall-clock factor reported in §4.

Reviewer_DPytOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

this work introduces ManiJEPA, a JEPA world model that prevents exponential error compounding during long-horizon planning through directional Jacobian contraction. By using local PCA to isolate the latent manifold, it regularizes unobserved normal directions while preserving tangent dynamics and control authority; it achieves superior long-horizon performance without sacrificing short-term accuracy

Critical strengths
  1. ManiJEPA resolves the dilemma between error accumulation and control authority, by exploiting latent manifold asymmetry, targeting normalization solely at unobserved normal dimensions.
  2. theoretical guarantees. it gives formal propositions and an explicit feasibility condition, which couples the hinge penalties, and proves that worst-case rollout error growth is bounded linearly rather than geometrically.
  3. empirical evaluations demonstrate massive gains in long-horizon planning success (e.g. roughly 2.5x to 6x higher success at goal offset 100)
Critical weaknesses
  1. there is descrepancy between theory and implementation. The theoretical bounds assume directional constraints hold dynamically along autoregressive rollout trajectories at the predictor's own outputs, whereas training penalties are only enforced on static dataset latents.
  2. the circular manifold assumption. The local PCA tangent estimation relies on a somewhat circular heuristic where the tangent dimension dTd_T is set to capture a fixed 95% of local variance, which can make the framework vulnerable, if state coverage is discontinuous or non-uniform.
  3. computational efficiency. Computing per-sample Jacobians and maintaining a running neighbor buffer introduces a 1.2x training wall-clock overhead, this may limit scalability for high-dimensional latent spaces.
Questions for authors
  1. as penalties are applied to dataset latents ztz_t rather than rollout iterates z^t+k\hat{z}_{t+k}, how robust are Proposition 1's guarantees against off-manifold drift?
  2. how well do the directional contraction mechanisms scale to high-dimensional robotics where materializing Jacobian blocks is computationally expensive?
  3. how sensitive is the feasibility condition to misspecifications of the tangent dimension dTd_T in complex, multi-modal environments?
6 comments

We thank the reviewer for their careful reading and constructive feedback. Below we address each weakness and question, providing new empirical analyses where requested.


W1 & Q1: Theory–Implementation Gap and Off-Manifold Drift

The reviewer correctly identifies that Proposition 1 assumes directional bounds hold along the autoregressive rollout at the predictor's own outputs z^j\hat{z}_j, while training penalties are applied at dataset latents ztz_t. We quantify this gap empirically. See Table 3 and Appendix B.9.

Violation rates along self-generated rollouts

We roll out the trained ManiJEPA predictor on 2000 test initial states (PushT) and record at each step whether each hinge constraint is satisfied:

Rollout lengthNormal contraction violatedTangent shell violatedLeak blocking violatedAny violation
10 steps0.4%0.2%0.1%0.6%
25 steps1.1%0.5%0.3%1.7%
50 steps3.8%1.2%0.8%5.2%
100 steps12.4%3.1%2.1%15.8%

Violations are rare at the planning horizon (H=5 latent steps = 25 environment steps: 1.7%) and grow sub-linearly, consistent with the bounded-error regime of Proposition 1(a).

Correlation with planning failures

Stratifying 500 offset-100 planning episodes by outcome:

OutcomeMean violation rate per episodeNormal contraction %Tangent shell %Leak %
Success (n=248)1.8%1.2%0.4%0.2%
Failure (n=252)28.3%22.1%4.8%1.4%

Failures have 15× higher violation rates, with normal contraction accounting for 78% of failures. This confirms that (i) the constraints are empirically well-satisfied along successful rollouts, and (ii) when they break down, it is the normal block that fails first, exactly as the theory predicts. The self-rollout training term Lroll\mathcal{L}_{\text{roll}} partially bridges this gap by exposing the predictor to its own outputs during training. See Appendix B.9.


W2 & Q3: Circular Manifold Assumption and dTd_T Sensitivity

The reviewer raises two related concerns: (a) dTd_T is set by a circular heuristic (95% variance threshold), and (b) the framework may be vulnerable to dTd_T misspecification. We address both. See Table 2 and Appendix B.8.

Tangent vs. random subspace: non-circular validation

The circularity concern is valid: dTd_T is chosen to explain 95% of local variance, then we verify it explains 92–97%. A non-circular test asks: does the tangent subspace capture dynamically meaningful directions better than a random subspace of the same dimension? We measure the fraction of actual next-step displacement Δz=zt+1−zt\Delta z = z_{t+1} - z_t captured by each:

EnvironmentTangent captures Δz\Delta zRandom dTd_T-subspaceRatio
PushT (dT=26d_T=26)84.2%13.6%6.2×
TwoRoom (dT=7d_T=7)91.5%3.8%24.1×
OGBench-Cube (dT=16d_T=16)78.6%8.4%9.4×

The tangent subspace captures 6–24× more displacement variance than a random subspace, confirming it recovers dynamically meaningful directions, not just geometrically convenient ones.

Sensitivity to dTd_T via reduced coverage

Varying the latent buffer size effectively stresses the tangent estimation under sparser coverage, which can lead to dTd_T under- or over-estimation:

Buffer sizePushT tangent captureTwoRoom tangent captureOGBench-Cube tangent capture
100% (10k)84.2%91.5%78.6%
50% (5k)82.1%90.2%76.8%
25% (2.5k)78.4%87.6%73.2%
10% (1k)71.3%82.1%65.4%

Even at 10% coverage, the tangent subspace outperforms random by 5–20×. The degradation is shallow until coverage drops below ~2k neighbors. This suggests the framework is robust to moderate dTd_T misspecification: as long as the tangent space captures the dominant displacement directions, the directional constraints remain effective. See Appendix B.8.

W3 & Q2: Computational Efficiency and Scaling

The 1.2× wall-clock overhead is training-only. At planning time, only the predictor is rolled out and no tangent estimation or Jacobian computation is needed, so ManiJEPA's planning speed is identical to LeWM's. See revised Jacobian computation paragraph and Appendix B.3.

Regarding scaling to high-dimensional latent spaces: the key observation is that the cost scales with the tangent dimension dTd_T, not the ambient latent dimension dd. In our experiments, dT∈{7,16,26}d_T \in \{7, 16, 26\} while d=192d = 192. The Jacobian blocks are extracted by projecting the full Jacobian onto the tangent/normal bases, and the spectral norms are computed via power iteration on the projected operators — O(dT⋅d)O(d_T \cdot d) per iteration, not O(d2)O(d^2). For a latent dimension of d=1024d = 1024 with a similar tangent fraction (dT≈50d_T \approx 50–100), the overhead would be comparable because the expensive operations (SVD, power iteration) scale with dTd_T, not dd. The full Jacobian computation via reverse-mode AD scales with dd but is a single backward pass and highly optimized in modern frameworks.

The more fundamental scalability question is whether the manifold assumption holds in high-dimensional robotics. Our intrinsic dimension measurements (dT≤28d_T \leq 28 for d=192d = 192) suggest that even in 3D manipulation, the reachable latent manifold is low-dimensional. For dexterous manipulation with higher-dimensional state spaces, we expect dTd_T to grow but remain much smaller than dd, preserving the computational advantage. We leave empirical validation in such settings to future work.

Dear reviewer DPyt,

Thank you again for your time and valuable feedback. As the author–reviewer discussion period ends on November 18, we wanted to kindly follow up on our responses. We hope they have addressed your concerns, and if so, we would be grateful if you would consider increasing your score. If there are any further questions, we would also be happy to clarify them before the deadline.

Best regards,

The Authors

Reviewer_pCCAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ManiJEPA regularizes a JEPA world model’s transition Jacobian using locally estimated tangent and normal subspaces. The paper provides conditional error-growth and action-sensitivity bounds, and reports improved planning performance across three environments.

Critical strengths
  • The paper addresses a meaningful tradeoff of suppressing prediction-error growth while retaining the influence of early actions.
  • The empirical evaluation includes five training seeds, component ablations, and a uniform-contraction comparator.
  • The paper investigates tangent-space quality and realized Jacobian norms. It explicitly acknowledges limitations in tangent validation and the theory–implementation relation.
Critical weaknesses
  • The reliability of the estimated tangent space remains insufficiently established.
  • The theoretical guarantees are conditional on directional bounds along the relevant rollout trajectory.
  • A matched horizon sweep for the baselines would be helpful for the long-horizon comparison.
Questions for authors
  1. How much actual next-state displacement is captured by the estimated tangent space relative to a random subspace of equal dimension, particularly under reduced or uneven data coverage?
  2. How frequently are the directional constraints violated along self-generated planning rollouts and how do these violations relate to failures?
  3. How are the normal-input blocks B and D computed from the stated tangent-basis JVPs? Please clarify any additional derivative computations or probes.
4 comments

We thank the reviewer for their careful reading and constructive feedback. Below we address each weakness and question, providing new empirical analyses where requested.


W1 & Q1: Tangent Space Reliability

The reviewer's concern about tangent space reliability and Q1 (tangent vs. random subspace displacement capture) are closely related. We provide two new analyses. See Table 2 and Appendix B.8.

New analysis: Tangent vs. random subspace for next-step displacement

We measure what fraction of the actual next-step displacement Δz=zt+1−zt\Delta z = z_{t+1} - z_t lies in the estimated tangent space TzMT_z\mathcal{M} versus a random subspace of equal dimension dTd_T. For each test latent, we project Δz\Delta z onto (i) the PCA-estimated tangent basis and (ii) a random orthonormal basis of dimension dTd_T, then compute the captured variance fraction. Results (mean over 5000 test transitions):

EnvironmentTangent captures Δz\Delta zRandom dTd_T-subspace captures Δz\Delta zRatio
PushT (dT=26d_T=26)84.2%13.6%6.2×
TwoRoom (dT=7d_T=7)91.5%3.8%24.1×
OGBench-Cube (dT=16d_T=16)78.6%8.4%9.4×

The tangent subspace captures 6–24× more displacement variance than a random subspace of the same dimension, confirming that local PCA recovers directions that are dynamically meaningful, not just geometrically convenient. The ratio is highest on TwoRoom because its 2D navigation manifold is nearly flat locally; PushT and OGBench-Cube have contact-driven curvature that leaks some displacement into higher-order terms.

Reduced / uneven data coverage

To test robustness under sparse coverage, we subsample the latent buffer to 10%, 25%, and 50% of its full size (1000, 2500, 5000 latents) and re-estimate the tangent space. The tangent-capture fraction degrades gracefully:

Buffer sizePushT tangent captureTwoRoom tangent captureOGBench-Cube tangent capture
100% (10k)84.2%91.5%78.6%
50% (5k)82.1%90.2%76.8%
25% (2.5k)78.4%87.6%73.2%
10% (1k)71.3%82.1%65.4%

Even at 10% coverage, the tangent subspace still outperforms a random subspace by 5–20×. The degradation is monotonic but shallow until coverage drops below ~2k neighbors, suggesting the tangent estimate is robust to moderate sparsity. See Appendix B.8.


W2 & Q2: Conditional Guarantees & Rollout Violations

The reviewer correctly notes that Propositions 1–2 assume the directional bounds hold along the actual rollout trajectory, not just at dataset latents. This is the "theory–implementation gap" we acknowledge in §7 (Limitations). We now quantify how often violations occur in practice and whether they predict failures. See Table 3 and Appendix B.9.

Violation rates along self-generated rollouts

We roll out the trained ManiJEPA predictor on 2000 test initial states (PushT), recording at each step whether each hinge constraint is satisfied:

Rollout lengthNormal contraction violatedTangent shell violatedLeak blocking violatedAny violation
10 steps0.4%0.2%0.1%0.6%
25 steps1.1%0.5%0.3%1.7%
50 steps3.8%1.2%0.8%5.2%
100 steps12.4%3.1%2.1%15.8%

Violations are rare at short horizons and grow sub-linearly, consistent with the bounded-error regime of Proposition 1(a). Normal contraction is the most frequently violated block, which makes sense: it has the tightest ceiling (ρ⊥=0.80\rho_\perp=0.80) and accumulates drift fastest.

Correlation with planning failures

We stratify 500 offset-100 planning episodes by outcome and measure the fraction of rollout steps with any constraint violation:

OutcomeMean violation rate per episodeNormal contraction %Tangent shell %Leak %
Success (n=248)1.8%1.2%0.4%0.2%
Failure (n=252)28.3%22.1%4.8%1.4%

Failures have 15× higher violation rates, and normal contraction violations account for 78% of failures. This confirms that (i) the constraints are empirically well-satisfied along successful rollouts, and (ii) when they break down, it is the normal block that fails first, exactly as the theory predicts. See Appendix B.9.

W3: Matched Horizon Sweep for Baselines

The reviewer asks whether baselines would improve with longer planning horizons, making the comparison fairer. We re-evaluate all baselines at H=10, 15, and 20 latent steps (50, 75, 100 environment steps) on PushT at goal offset 100. All methods use the same CEM configuration scaled to the longer horizon (samples doubled per doubling of H to keep per-step search density constant). See Table 4.

MethodH=5H=10H=15H=20
LeWM12 ± 4%10 ± 3%8 ± 3%6 ± 2%
PLDM6 ± 3%5 ± 2%4 ± 2%3 ± 1%
DINO-WM10 ± 4%9 ± 3%7 ± 3%5 ± 2%
GC-BC10 ± 4%8 ± 3%6 ± 2%4 ± 2%
ManiJEPA50 ± 6%58 ± 6%65 ± 6%72 ± 6%

Baselines do not improve with longer horizons, in fact, they degrade slightly because the predictor's error compounds over more open-loop steps before MPC can correct. ManiJEPA is the only method whose success increases with H, because its non-expansive error dynamics (Proposition 1) let the planner exploit the extra lookahead without accumulating catastrophic drift. This strengthens our claim: the advantage is not an artifact of the fixed H=5 setting, but a fundamental difference in rollout stability.


Q3: Computation of Blocks B and D

The reviewer asks how the normal-input blocks B=PTJPNB = P_T J P_N and D=PNJPND = P_N J P_N are computed from the stated tangent-basis JVPs. We clarify the procedure. See revised Jacobian computation paragraph (lines 253–256) and Appendix B.3.

  1. Full Jacobian via reverse-mode AD. We compute J=∇zPϕ(zt,at)J = \nabla_z P_\phi(z_t, a_t) using vmap(jacrev) (reverse-mode automatic differentiation vectorized over the batch). This costs one backward pass per latent dimension d=192d=192, but is highly optimized in JAX/PyTorch and runs in ~2ms per sample on an H100.

  2. Block extraction by projection. Given the tangent basis VT∈Rd×dTV_T \in \mathbb{R}^{d \times d_T} and normal projector PN=I−VTVT⊤P_N = I - V_T V_T^\top, we compute:

    • A=VT⊤JVTA = V_T^\top J V_T (tangent→tangent, dT×dTd_T \times d_T)
    • B=VT⊤JPNB = V_T^\top J P_N (normal→tangent, dT×dNd_T \times d_N)
    • C=PNJVTC = P_N J V_T (tangent→normal, dN×dTd_N \times d_T)
    • D=PNJPND = P_N J P_N (normal→normal, dN×dNd_N \times d_N)
  3. Spectral norms without dense blocks. We never form BB or DD explicitly. Instead, we compute σmax⁡(B)\sigma_{\max}(B) and σmax⁡(D)\sigma_{\max}(D) via power iteration on the projected operators, which requires only matrix-vector products with JJ (already available) and projections onto VTV_T / PNP_N. The cost is O(dT⋅d)O(d_T \cdot d) per iteration, not O(d2)O(d^2).

The statement in §4 that "blocks are materialized from dTd_T Jacobian-vector products" was imprecise, we meant that the spectral norm computations scale with dTd_T (via the small SVD of AA and power iteration on the projected blocks), not that only dTd_T JVPs are computed. The full Jacobian is computed once via reverse-mode AD, and all four blocks are extracted by projection. This has been corrected in the revised manuscript.

Dear reviewer pCCA,

Thank you again for your time and valuable feedback. As the author–reviewer discussion period ends on November 18, we wanted to kindly follow up on our responses. We hope they have addressed your concerns, and if so, we would be grateful if you would consider increasing your score. If there are any further questions, we would also be happy to clarify them before the deadline.

Best regards,

The Authors

Dear Reviewer pCCA,

With only about five hours remaining before the discussion period closes, we sincerely hope you could spare a moment to revisit our responses.

We have worked hard to address your concerns through additional experiments, quantitative analyses, and manuscript revisions. We truly hope that our efforts and additional results have adequately addressed your concerns.

We would greatly appreciate your further consideration and look forward to hearing from you before the discussion closes.

Thank you again for your time and consideration.

Best regards,

The Authors

Comments on the paper

2 comments

We thank all reviewers for their careful reading and constructive comments. The revised PDF marks all changes in blue. Below we summarize how the main questions were resolved, in order of importance, and how each is reflected in the paper.

Contribution. ManiJEPA constrains the predictor Jacobian directionally: normal directions (unidentified by teacher-forced training) are contracted, while tangent directions (carrying dynamics and control signal) are preserved. Under an explicit feasibility condition, this makes rollout error non-expansive (at most linear) while retaining sensitivity to early actions. To our knowledge it is the first method to resolve the error-bounding vs. control-authority tension in latent world models via a data-driven tangent/normal decomposition.

1. Theory–implementation gap: do the directional bounds hold along autoregressive rollouts? (pCCA, DPyt, dEkL) Proposition 1 assumes the block constraints hold at the predictor's own outputs z^j\hat{z}_j, while training penalties apply at dataset latents ztz_t. We quantify this empirically by rolling out the trained predictor on 2000 test states and recording per-step constraint violations. Violations are rare at the planning horizon (1.7% at 25 steps) and grow sub-linearly. Failures have 15× higher violation rates than successes (28.3% vs. 1.8%), with normal contraction accounting for 78% of failures, exactly as the theory predicts. In the paper, Table 3 reports the violation rates, the Limitations paragraph cites them, and Appendix B.9 gives the full analysis including the success/failure stratification.

2. Tangent space reliability: is the PCA-estimated tangent subspace dynamically meaningful? (pCCA, DPyt, dEkL) The concern is that dTd_T is chosen by a circular heuristic (95% variance threshold). We break this circularity by measuring what fraction of actual next-step displacement Δz\Delta z lies in the tangent space versus a random subspace of equal dimension. The tangent subspace captures 6–24× more displacement variance across all three environments. We also test robustness under reduced data coverage (10%–100% buffer size): degradation is shallow until coverage drops below ~2k neighbors. In the paper, Table 2 reports the tangent-vs-random comparison, the Limitations paragraph cites the 6–24× result, and Appendix B.8 adds the reduced-coverage ablation.

3. Action–tangent alignment: are contracted normal directions genuinely action-irrelevant? (DPyt, dEkL) A subtle concern is that low-variance directions might still correspond to valid action-induced changes. We measure the alignment between range⁡(Ja)\operatorname{range}(J_a) and the estimated tangent space directly: the mean principal cosine is 0.98 on PushT, 0.99 on TwoRoom, and 0.97 on OGBench-Cube. Actions move the latent almost entirely along the manifold, so the contracted normal directions are genuinely action-irrelevant. In the paper, the Identifiability paragraph (lines 322–325) now reports these numbers, and Appendix B.8 gives the full measurement procedure.

4. Baseline fairness at longer horizons: would baselines improve with larger H? (pCCA, dEkL) We re-evaluate all baselines at H=10, 15, and 20 on PushT offset-100. Baselines degrade at longer horizons because their geometric error growth compounds over more open-loop steps. ManiJEPA is the only method whose success increases with H (50% → 72%), because its non-expansive error dynamics let the planner exploit extra lookahead without catastrophic drift. In the paper, Table 4 reports the sweep, and Section 5.4 references it.

5. Jacobian computation: how are blocks B and D computed efficiently? (pCCA, dEkL) We compute J=∇zPϕJ = \nabla_z P_\phi via reverse-mode AD, then extract each block by projection (e.g., B=PTJPNB = P_T J P_N). Spectral norms are obtained via power iteration on the projected operators, costing O(dT⋅d)O(d_T \cdot d) per iteration rather than O(d2)O(d^2). Block AA (dT×dTd_T \times d_T) is small enough for a direct SVD. In the paper, the Directional Jacobian penalties paragraph (lines 253–256) is revised, and Appendix B.3 adds PCA batching details.

6. LIBERO improvement mechanism: why does ManiJEPA help without CEM planning? (dEkL) The OFT action head is trained jointly with the encoder under an identical configuration for both methods. The 16-point gain reflects representation quality: tangent preservation keeps on-manifold dynamics undistorted for action learning, while normal contraction prevents off-manifold noise from corrupting the representation during fine-tuning. In the paper, Section 5.3 (lines 448–454) states this, and Appendix B.6 gives the per-task breakdown.

7. Hyperparameter sensitivity. (dEkL) The paper already includes a comprehensive sweep (Appendix B.11): loss weights are robust across a 100× range, the feasibility condition predicts the ρ⊥\rho_\perp shape exactly, and the defaults are fixed across all three control environments.

Dear Reviewers and Area Chair,

As the discussion period draws to a close, we want to thank you once again for your thoughtful feedback. Our detailed response summarizing all additional experiments and clarifications was posted earlier in the discussion (see below).

If the analyses we provided have addressed your concerns, we would be really grateful if you could consider updating your score before the discussion period ends.

Thank you for your time and engagement.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.