acceptodds
Under review as a conference paper at ICLR 2027

Control-Geometry Straightening for Sampling-Based Latent Planning

Abstract

Joint-embedding predictive architectures enable planning with latent world models, but accurate transition prediction alone does not ensure that the planning objective is easy to optimize. We introduce Control-Geometry Straightening (CGS), a single auxiliary loss that learns planner-friendly representations by directly straightening control geometry for sampling-efficient planning. CGS matches pairwise cosine similarities among actions to those among corresponding latent differences only using local transitions from pixel–action pairs. The loss can be applied across world-model architectures using end-to-end learned or pretrained representations. Under linear-dynamics, our theoretical analysis connects this objective to temporal straightening and more balanced terminal-cost curvature across the full planning horizon, yielding finite-budget guarantees for MPPI, local contraction results for CEM, and convergence bounds for gradient descent. Across four control environments and multiple planners, CGS improves planning with fewer sampled candidates and refinement steps, achieving success-rate gains up to and percentage points over LeWorldModel (LeWM) and its temporal-straightening variant (LeWM+TS), respectively, with sampling-based planners using candidates per update. Probes, comparisons with DINO-WM architecture, and planner-side ablations clarify how latent motion organization, state dependence, and dynamical context shape planning behavior. Straightening control geometry thus makes good action sequences easier to find under limited planning budgets.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
World Models, Representation Learning, Sampling-Based Planning, Model Predictive Control, Latent Dynamics, Control Geometry

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_F2HGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Control-Geometry Straightening (CGS), an auxiliary loss that matches pairwise cosine similarities among actions to those among corresponding latent differences. The goal is to learn planner-friendly representations where the geometry exposed to action optimization is well-conditioned. Under linear dynamics, the paper provides theoretical analysis connecting CGS to temporal straightening and balanced terminal-cost curvature, yielding finite-budget guarantees for MPPI, local contraction results for CEM, and convergence bounds for gradient descent. Experiments on PushT, Cube, TwoRooms, and Reacher show success-rate gains up to 20 and 12.6 percentage points over LeWM and LeWM+TS, respectively, with sampling-based planners using 128 candidates per update.

Critical strengths

a. Making representations easier to plan in, not just accurate, is an important and underexplored direction. The CGS loss is simple to implement and applies to any world model architecture. The probing analysis (Table 2) provides valuable insights into when action-only vs. state-conditioned references matter.

b. The paper provides finite-budget guarantees for MPPI, local contraction results for CEM, and convergence bounds for gradient descent under linear dynamics. The analysis connecting local geometry (CGS loss) to global planning curvature (planning Hessian) is rigorous and informative. The ε\varepsilon-CGS bounds (Corollary 4.3) and the Hessian perturbation bound (Theorem 4.4) are particularly well-done.

c. The paper evaluates multiple planners (MPPI, CEM, GD), budget sweeps (K = 1 to 256), ablations (ramp fraction, history length, SIGReg weight, CGS weight), and probing analyses. The DINO-WM comparison in the appendix, while mixed, shows that CGS can apply to different architectures.

Critical weaknesses

a. CGS is a pairwise cosine similarity matching loss, which is a standard technique from knowledge distillation (Tung & Mori, 2019). The paper applies this to actions vs. latent differences, which is a reasonable application but not a fundamental algorithmic advance. The "control geometry" terminology makes it sound more novel than it is. The paper would benefit from a clearer discussion of what is genuinely new vs. what is a straightforward application of existing techniques.

b. The theoretical analysis is limited to linear dynamics, which is a significant gap. The paper provides rigorous guarantees under linear dynamics but experiments on nonlinear systems. While the paper acknowledges this limitation, it doesn't provide any analysis of when or why the linear theory carries over to nonlinear practice. A discussion of the conditions under which the linear approximation is valid (e.g., local linearity, small perturbations) would strengthen the theoretical contribution.

c. The paper compares against LeWM and LeWM+TS, but doesn't compare against other objectives that might also improve planning geometry (e.g., contrastive learning, inverse dynamics, DeepMDP, CURL). This makes it unclear whether CGS is the best approach or just one of many viable options. A comparison with at least one alternative objective would strengthen the empirical contribution.

d. On Reacher, CGS provides minimal improvement for MPPI (65.3% → 66.7%) and actually decreases CEM performance (83.3% → 80.7%). The paper attributes this to state-dependence but doesn't provide a deeper analysis. Understanding when CGS fails is as important as understanding when it succeeds.

Suggestion: At 50+ pages, some sections could be condensed. The core contribution could be presented more concisely. The extensive appendix, while thorough, makes the paper feel bloated.

Questions for authors
  1. Can you provide a comparison with other representation learning objectives (e.g., contrastive learning, inverse dynamics, DeepMDP) to establish whether CGS is the best approach or just one of many viable options?

  2. Can you provide a more detailed analysis of when CGS fails (e.g., on Reacher)? What properties of the task or dynamics make CGS less effective?

  3. Can you bridge the gap between the linear theory and nonlinear practice? For example, can you provide empirical evidence that the planning Hessian is better-conditioned under CGS on the nonlinear tasks?

  4. Can you provide more cross-architecture validation in the main text? The DINO-WM results in the appendix are mixed, and it would be valuable to understand when CGS helps vs. hurts with different backbones.

7 comments

Thank you for the constructive review and for recognizing the importance of planner-friendly representations, the geometry-to-planning analysis, and the breadth of our evaluation. CGS identifies a local action–transition constraint that shapes finite-horizon planning curvature, with a linear-model analysis connecting this geometry to finite-budget planning. It uses a single auxiliary loss on observation–action transitions without reward, goal, or image-reconstruction supervision for training. Your questions helped us clarify this contribution and strengthen its empirical support: the revision adds alternative-objective comparisons, nonlinear curvature measurements, deeper Reacher diagnostics, and expanded DINO-WM results in the main text.

W1: “The paper would benefit from a clearer discussion of what is genuinely new vs. what is a straightforward application of existing techniques.”

Thank you for asking us to clarify our contribution. CGS translates the goal of improving JEPA planning-Hessian geometry into a local training target, with theoretical connections to finite-budget optimization and empirical improvements in planning efficiency.

Our starting point was the insight from TS (Wang et al., 2026b) that near-identity latent dynamics can improve planning. For zt+1=Azt+Batz_{t+1}=Az_t+Ba_t, however, even A=IA=I leaves the anisotropy of B⊤BB^\top B unconstrained. We therefore worked backward from the desired finite-horizon planning curvature to identify a local geometric surrogate for its elegance and computation efficiency, avoiding direct Hessian regularization through an HH-step predictor rollout.

Under the stated linear-model, fixed-radius, and coverage assumptions, our analysis shows that local action–transition Gram matching jointly controls temporal drift, drift–control coupling, and action anisotropy: A≈IA\approx I, (A−I)⊤B≈0(A-I)^\top B\approx 0, and B⊤B≈cIB^\top B\approx cI. These constraints yield finite-horizon curvature bounds and corresponding MPPI, CEM, and GD guarantees (Sec. 4; Appendix D). The practical cosine objective expresses this local angular relation: latent transitions remain aligned when actions align and turn when actions turn. The pairwise similarity-matching form has precedents in similarity-preserving knowledge distillation (Tung & Mori, 2019). Our contribution is to identify local action–transition matching as a training target for shaping planning geometry and establish its connection to finite-horizon curvature and finite-budget optimization.

The original submission evaluated this principle across four environments and three planners, with sampling-based success gains up to 20.0 and 12.6 percentage points over LeWM and TS at K=128K=128. The revision strengthens this evidence through direct nonlinear-Hessian measurements, alternative-objective comparisons, expanded DINO-WM evaluation, and Reacher reference/history analyses, clarifying both the mechanism and its applicability.

The name Control-Geometry Straightening reflects this extension of Temporal Straightening: from straightening latent trajectories to shaping how action effects accumulate into the finite-horizon planning Hessian. The revised Sec. 4 (pp. 4–7) makes the inherited framework and our additional contributions explicit.

W2: “The theoretical analysis is limited to linear dynamics, which is a significant gap.”

Q3: “Can you bridge the gap between the linear theory and nonlinear practice? For example, can you provide empirical evidence that the planning Hessian is better-conditioned under CGS on the nonlinear tasks?”

Thank you for asking us to clarify the connection between our linear theory and nonlinear practice. We now provide both explicit conditions for the local linear interpretation and measurements of the actual nonlinear predictor's planning curvature (Remark 4.8, pp. 6–7; Fig. 5, p. 8; Table 13, p. 32).

For smooth dynamics, a local linear approximation captures first-order control responses when candidate rollouts remain in a neighborhood where the state and action Jacobians vary little. For the nonlinear terminal prediction F^H(a)\widehat{F}_H(\mathbf{a}), let eH(a)=F^H(a)−zge_H(\mathbf{a})=\widehat{F}_H(\mathbf{a})-z_g and JH=∂eH/∂aJ_H=\partial e_H/\partial\mathbf{a}. The exact Hessian of C(a)=12∥eH(a)∥22C(\mathbf{a})=\frac{1}{2}\lVert e_H(\mathbf{a})\rVert_2^2 is

=\underbrace{J_H^\top J_H}_{G_H^{\mathrm{GN}}} +\sum_j e_{H,j}(\mathbf{a})\nabla_{\mathbf{a}}^2 e_{H,j}(\mathbf{a}).$$ The first-order response determines Gauss–Newton curvature. It also approximates full curvature when the residual-weighted second-order term is small relative to it, as can occur with weak rollout nonlinearity or small terminal residuals and bounded second derivatives. **We directly measure Gauss–Newton and full-Hessian conditioning in the nonlinear transformer predictor used for planning**, testing whether the geometric benefits identified by our linear analysis also appear in practice. The table reports effective condition numbers computed from the largest $d_a$ singular values. Entries follow **LeWM / TS / CGS** for PushT, Cube, and TwoRooms, and **LeWM / TS / CGS-C / CGS-S** for Reacher. CGS-C and CGS-S use concatenated and summed action references, respectively; their comparison is discussed below in our response to **W4/Q2**. | Metric ↓ | PushT | Cube | TwoRooms | Reacher | | --- | --- | --- | --- | --- | | $\kappa_{\mathrm{eff}}^{d_a}(G_H^{\mathrm{GN}})$ | 2.24 / 2.18 / **1.65** | 680.84 / 138.19 / **25.78** | 3.27 / 2.03 / **1.78** | **1.23** / 1.60 / 1.60 / 1.31 | | $\kappa_{\mathrm{eff}}^{d_a}(G_H^{\mathrm{full}})$ | 7.52 / 7.69 / **5.82** | 38.15 / **7.56** / 14.92 | 17.78 / 7.57 / **7.18** | 11.55 / 7.66 / 8.97 / **4.12** | | $\kappa_{\mathrm{eff}}^{d_a}(G_H^{\mathrm{full,w}})$ | 2.80 / 2.87 / **2.06** | 243.68 / 102.87 / **56.00** | 3.26 / 2.76 / **1.92** | **1.29** / 1.61 / 1.63 / 1.31 | The superscript $w$ denotes evaluation at expert action sequences plus Gaussian noise with standard deviation $0.05$ (Eq. 29). In this near-expert regime, the full planning Hessian more closely matches its Gauss–Newton approximation. **CGS improves both Gauss–Newton and full-Hessian conditioning on PushT and TwoRooms.** On Cube, it substantially improves Gauss–Newton conditioning, while TS has better full-Hessian conditioning at the standard probe points; CGS has the best full-Hessian conditioning near perturbed expert actions. On Reacher, CGS-S yields the clearest full-Hessian improvement, which will be discussed in our response to **W4/Q2** in detail. Fig. 5 relates these measurements to MPPI success, with CEM/GD results in Appendix C.5.4. They support curvature balance as a contributing mechanism and identify where other limitations matter. Similar condition numbers near expert actions do not, by themselves, establish Gauss–Newton dominance; that would require comparing both Hessian terms at the same action sequence.

W3: “The paper compares against LeWM and LeWM+TS, but doesn't compare against other objectives that might also improve planning geometry (e.g., contrastive learning, inverse dynamics, DeepMDP, CURL).”

Q1: “Can you provide a comparison with other representation learning objectives (e.g., contrastive learning, inverse dynamics, DeepMDP) to establish whether CGS is the best approach or just one of many viable options?”

Thank you for suggesting this comparison. We added both inverse-dynamics and temporal-contrastive comparisons under the same JEPA training and planning setup; CGS achieves the highest mean success across all three planners on PushT. The definitions and results are now in Appendix C.5.6 (pp. 33–34), Table 14.

Inverse dynamics (LDAD). Following Delta-JEPA (Zhang et al., 2026b), a training-only decoder reconstructs actions from latent displacements:

=\frac{1}{BTd_a}\sum_{b=1}^{B}\sum_{t=0}^{T-1} \left\\|D_{\psi}(z_{b,t+1}-z_{b,t})-a_{b,t}\right\\|_2^2.$$ We test weight 10, following Delta-JEPA, and the larger value 100. Accurate action decoding preserves action information but **does not itself ensure that latent similarities reflect action similarities or that planning curvature is well conditioned**. **Temporal contrastiveness (TC).** Following the comparison in TS ([Wang et al., 2026b, Appendix B.5](https://arxiv.org/html/2603.12231v3)), we use $\mathcal{L}_{\mathrm{TC}}=\operatorname{InfoNCE}(z_{\mathrm{pos}},z_{\mathrm{neg}})$, with other frames within a temporal window of length $N$ as positives and frames outside that window in the same trajectory as negatives. This encourages temporal proximity in the representation, but **can incorrectly separate similar states revisited later in a trajectory**. The table reports mean success (%) over three evaluation seeds, with 50 episodes each. CEM/MPPI use $K=128$, $I=30$; GD uses 100 steps. We retain the LeWM ([Maes et al., 2026](https://arxiv.org/abs/2603.19312)) backbone, prediction loss, and SIGReg objective. | Method | Weight | CEM | MPPI | GD | | --- | --- | --- | --- | --- | | LeWM | — | 89.3 | 64.0 | 85.3 | | TS | 0.01 | 86.7 | 64.7 | 82.7 | | CGS | 0.01 | **92.0** | **77.3** | **89.3** | | LDAD | 10 | 87.3 | 70.7 | 83.3 | | LDAD | 100 | 80.7 | 58.0 | 70.0 | | TC | 0.001 | 90.7 | 63.3 | 88.0 | | TC | 0.01 | 84.0 | 53.3 | 76.7 | LDAD improves MPPI, and TC improves CEM and GD at the smaller weight, showing that these objectives can also help. **CGS achieves the highest mean across all three planners**, and larger weights degrade both alternatives. These results support explicitly shaping control geometry beyond action recoverability or temporal proximity, consistent with our analysis of balanced planning curvature. DeepMDP ([Gelada et al., 2019](https://proceedings.mlr.press/v97/gelada19a.html)) jointly models rewards and latent transitions, while CURL ([Laskin et al., 2020](https://proceedings.mlr.press/v119/laskin20a.html)) combines contrastive representation learning with off-policy RL. Our added baselines isolate inverse-dynamics and temporal-contrastive objectives within reconstruction-free, reward-free JEPA training. This provides a controlled comparison of auxiliary representation objectives under the same supervision and model class.
Reviewer_AZi5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a new regularization for world models called Control-Geometry Straightening (CGS). It extends the temporal straightening loss, which encourages the cos between consecutive latent displacements, to align it with cosine similarities among actions. Its major experiments show improvement compared to LEWM and LEWM+TS for some setups.

Critical strengths
  • The idea is well-motivated: using action provides additional information that original temporal straigtening doesnt provide
  • The method is simple and easy to use
  • Detailed experiment results
Critical weaknesses
  • The major concern is that the action similarity can be a poor proxy for transition similarity. The same action can have very different effect depending on factors like momentum and contact.
  • The empirical results are mixed: In Figure 4, the proposed method is quite close to the TS baseline for 3/4 environments. In Table1, the gain of the proposed method is also not very significant. It would be great if the authors could explain and show under which situations the proposed method will be better or worse.
  • The theory section would be clearer if it explicitly distinguish the framework and results inherited from TS from the new theoretical contributions.
  • (writing) It would be better to show the performance of different models grouped by planner. The current presentation makes it hard to compare numbers.
Questions for authors

see weakness; would be happy to raise scores if all concerns are addressed

6 comments

Thank you for the constructive review and for recognizing CGS's motivation, simplicity, and detailed evaluation. CGS shapes finite-horizon planning geometry using only local pixel–action data, without reward, goal, or image-reconstruction supervision. Its contribution connects action–transition matching to control-geometry bounds and finite-budget sampling-based planning under stated linear-model assumptions.

W1: “The major concern is that the action similarity can be a poor proxy for transition similarity. The same action can have very different effect depending on factors like momentum and contact.”

Thank you for this comment, which motivated a deeper experimental investigation of Reacher. We agree that momentum and contact can make action similarity a poor transition reference. Our new results suggest two contributors to planning failures on Reacher: difficulty learning favorable control geometry under torque-driven dynamics, and missing initial velocity information when planning starts from a single image. Both TS (Wang et al., 2026b) and default CGS show modest, planner-dependent gains over LeWM (Maes et al., 2026) in this setting. We have added these analyses to Sec. 6 (pp. 9–10), Fig. 6 (p. 9), and Appendix C.5.5 (pp. 32–33).

CGS encourages pairwise angular agreement, cos⁡(Δzi,Δzj)≈cos⁡(ai,aj)\cos(\Delta z_i,\Delta z_j)\approx\cos(a_i,a_j), as an auxiliary objective alongside prediction. It does not require actions to be displacements or uniquely determine transitions. The theoretical model Δzt=(A−Id)zt+Bat\Delta z_t=(A-I_d)z_t+Ba_t allows a transition to contain both an action effect and state-dependent drift. Appendix D.7 (pp. 47–48) analyzes how control geometry can still be shaped when this drift must be retained. The usefulness of the reference therefore depends on how informative local action relations are about transition geometry.

The physical probes in Tables 2 and 9 (pp. 9 and 27) show that Reacher's torque-driven motion depends strongly on configuration and velocity: adding state raises fingertip-displacement prediction R2R^2 from −0.006-0.006 to 0.9830.983. This motivates examining whether a different summary of the same torque inputs provides a more useful geometric reference. Default CGS-Concat (CGS-C) preserves the time-ordered structure of each five-action block, whereas CGS-Sum (CGS-S) sums the standardized primitive torques, capturing cumulative actuation and cancellation. For short blocks with slowly varying torques and approximately constant local dynamics, this impulse-like quantity approximates, up to scale, the time-weighted actuation contributing to displacement (Appendix C.5.5, Eq. 28).

We evaluate the resulting planning geometry through autoregressive predictor rollouts, differentiating with respect to the candidate action sequence while fixing the observed context and goal. For C(a)=12∣eH(a)∣22C(\mathbf a)=\frac12\\|e_H(\mathbf a)\\|_2^2, where eH(a)=F^H(a)−zge_H(\mathbf a)=\widehat F_H(\mathbf a)-z_g and JH=∂eH/∂aJ_H=\partial e_H/\partial\mathbf a, the exact Hessian is

=\underbrace{J_H^\top J_H}_{G_H^{\mathrm{GN}}} +\sum_j e_{H,j}(\mathbf a)\nabla_{\mathbf a}^2 e_{H,j}(\mathbf a).$$ The comparison below is included in Table 13 (p. 32). It reports effective condition numbers of Gauss-Newton planning Hessian $\kappa_{\mathrm{eff}}^{d_a}(G_H^{\mathrm{GN}})$ (under linear dynamics) and full planning Hessian $\kappa_{\mathrm{eff}}^{d_a}(G_H^{\mathrm{full}})$ over the leading $d_a$ directions. | Effective condition number ↓ | LeWM | TS | CGS-C | CGS-S | | --- | ---: | ---: | ---: | ---: | | $\kappa_{\mathrm{eff}}^{d_a}(G_H^{\mathrm{GN}})$ | **1.23** | 1.60 | 1.60 | 1.31 | | $\kappa_{\mathrm{eff}}^{d_a}(G_H^{\mathrm{full}})$ | 11.55 | 7.66 | 8.97 | **4.12** | TS and default CGS (CGS-C) modestly improve full-Hessian conditioning over LeWM. CGS-S achieves a substantially larger improvement, reducing the effective condition number to **4.12**, compared with **11.55, 7.66, and 8.97** for LeWM, TS, and CGS-C, respectively. Its Gauss–Newton conditioning remains close to LeWM's already near-isotropic level. Thus, changing the action reference yields its clearest benefit in full-Hessian conditioning. To assess how better conditioning translates into planning, we include **transition-CGS as a positive control**, using privileged physical state information to construct realized whole-arm displacements as the training reference (Appendix C.5.5, Eq. 30). Planning still uses images and actions. With **naive inputs**, the default planning paradigm with a single-frame input at each replanning call, CGS-S performs close to this control and achieves higher mean MPPI and CEM success, but substantial headroom remains: every solve starts from one image, which leaves initial velocity unobserved.

We therefore test whether observation memory helps realize the benefit of improved geometry. Online history retains the last three actual observations and their connecting executed actions from the first execution cycle as memory for replanning. Privileged history additionally supplies preceding dataset observations and actions at the first solve; later solves use online memory. Each trained checkpoint remains fixed.

The table reports mean success (%) over three evaluation seeds, with 50 episodes each (Fig. 6, p. 9; protocols in Appendix C.5.5, pp. 32–33). CEM/MPPI use K=128K=128, I=30I=30; GD uses 100 steps.

PlannerHistoryLeWMTSCGS-CCGS-STransition-CGS
MPPINaive65.364.066.772.770.0
MPPIOnline69.372.768.780.079.3
MPPIPrivileged68.774.772.785.386.0
CEMNaive83.381.380.787.384.0
CEMOnline92.794.092.096.098.0
CEMPrivileged96.097.398.798.799.3
GDNaive76.781.382.081.382.7
GDOnline81.388.089.393.394.0
GDPrivileged84.089.390.794.095.3

With observation memory, CEM approaches near-perfect success, while MPPI and GD improve substantially. CGS-S outperforms LeWM, TS, and CGS-C under MPPI and GD in both history settings and remains close to transition-CGS. An action-only reference can therefore achieve planning performance comparable to the physical-displacement reference in this setting, supporting CGS's ability to learn useful planning geometry without privileged motion labels.

Returning to naive inputs, the remaining failures reveal a gap between approaching the goal and completing it precisely. Fig. 6(d) reports the fraction of failed rollouts reaching joint-angle error e<0.10e<0.10 rad; success requires e<0.05e<0.05 rad:

PlannerLeWMTSCGS-CCGS-S
MPPI79.2%78.7%76.7%82.9%
CEM84.6%85.7%89.5%95.0%
GD38.9%42.4%45.5%60.7%

Despite improved full-Hessian conditioning, CGS-S still falls short of perfect success under naive single-frame inputs. Its higher near-goal failure fraction, together with the gains from adding observation history, suggests that better geometry helps planning approach the goal, while missing velocity information limits precise completion.

W2: “The empirical results are mixed … It would be great if the authors could explain and show under which situations the proposed method will be better or worse.”

Thank you for requesting a clearer explanation of this variation. TS is a strong baseline, and CGS extends its straightening principle with more flexible, action-dependent angular targets: latent transitions can remain aligned when actions align and turn when actions turn. Its additional benefit depends on the environment's control structure, how much geometry TS already captures, and the planner's remaining limitations. The revised Secs. 5.2–6 (pp. 7–10), Fig. 5 (p. 8), and Table 13 (p. 32) provide the corresponding performance and geometry analyses.

PushT. Pushing requires changes in control direction, while LeWM already exhibits strong implicit straightening, leaving limited additional benefit from TS (Fig. 10, p. 26). CGS captures these action-related turns and improves both Gauss–Newton and full-Hessian conditioning, producing clearer planning gains.

Cube. Sustained, similarly directed grasp-and-translate controls favor TS, which produces a pronounced temporal straightening effect (Fig. 10, p. 26) and substantial planning gains over LeWM. CGS further improves Gauss–Newton conditioning, while TS retains better full-Hessian conditioning (Fig. 5, p. 8; Table 13, p. 32). This strong TS baseline and the mixed additional curvature improvements help explain CGS's more modest gains over TS.

TwoRooms. Simple navigation with small local changes in action direction makes CGS's targets close to TS's alignment target. Both achieve similar improvements in planning-Hessian conditioning, while sampling-based success approaches saturation, leaving limited additional benefit for CGS.

Reacher. As discussed in W1, torque-reference mismatch and missing velocity information constrain the gains. The reference and observation-memory experiments above show how addressing these two limitations improves planning.

W3: “The theory section would be clearer if it explicitly distinguish the framework and results inherited from TS from the new theoretical contributions.”

Thank you for this suggestion. We have modified the theory part Sec. 4 (pp. 4–5) and made the attribution explicit throughout the analysis. We inherit TS's linear latent-dynamics framework, its connection between near-identity transitions and planning conditioning for gradient-based planner, and its controllable-subspace viewpoint. The general spectral tools for gradient-descent convergence are also standard. Here we restate our CGS-specific contributions as below:

  1. Matching loss → joint control geometry. Proposition 4.2 and Corollary 4.3 show that CGS jointly controls temporal drift, drift–control coupling, and action anisotropy, yielding the ε\varepsilon-CGS bounds. Temporal straightness alone does not impose this joint control.
  2. Joint geometry → finite-horizon curvature. Theorem 4.4 bounds the deviation of the planning Hessian from an isotropic projector by ξH(ε)\xi_H(\varepsilon), incorporating the additional drift–control constraint. Remark 4.5 and Appendix D.2.2 specify the leading-dad_a guarantee for low-dimensional actions and the conditions needed to cover all controllable directions.
  3. Curvature → sampling-based planning. Theorem 4.6 and Appendix D.4 derive population MPPI contraction and finite-budget planning-error bounds. Appendix D.5 gives local CEM contraction results; Appendix D.6 applies the CGS spectrum to gradient-descent convergence, extending TS's conditioning analysis to convergence analysis for gradient-based planners.

The empirical evidence follows this chain: Fig. 1 visualizes predictor-response geometry; Fig. 3 tests MPPI contraction and finite-sample progress using measured predictor geometry; Fig. 5 and Table 13 extend the diagnostics to nonlinear planning curvature and success. The linear assumptions and the local connection to nonlinear dynamics are now explicitly discussed in Remark 4.8 (pp. 6–7). This separates the inherited analytical framework, our new bounds, and their empirical support.

W4: “It would be better to show the performance of different models grouped by planner. The current presentation makes it hard to compare numbers.”

Thank you for this helpful suggestion. Table 1 (p. 8) now groups methods within each planner, with CGS rows shaded and the best means marked in bold. It also integrates the expanded DINO-WM (Zhou et al., 2025) evaluation across all four environments and all three planners, using both CLS and Patch representations. CGS achieves higher mean success than TS in all 12 CLS settings. With Patch representations, CGS is competitive under CEM and MPPI, while TS retains advantages in several GD settings.

Summary of manuscript revisions

We have clarified the theoretical contribution and strengthened the empirical evidence, while reducing the full manuscript from 55 to 48 pages, including references and appendices. Sec. 4 distinguishes the framework inherited from TS from our contributions and discusses the connection between linear and nonlinear dynamics (Remark 4.8, p. 6). Table 1 (p. 8) presents the expanded cross-architecture evaluation, while Sec. 6 adds nonlinear planning-Hessian conditioning (Fig. 5, p. 8) and deeper Reacher analyses (Fig. 6, p. 9). Appendix C reorganizes the supporting protocols and analyses, alongside new alternative-regularizer comparisons (Table 14, p. 33) and an expanded CGS-weight ablation (Fig. 9, p. 25). We also refined the abstract and introduction to reflect the clarified contributions and expanded empirical evidence.

Reviewer_FwVAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Control-Geometry Straightening (CGS), which is an auxiliary training loss for latent world models that aims to make good action sequences easier to find with fewer sampled candidates and refinement steps. CGS trains the representation so that the pairwise cosine similarities among consecutive latent changes match those among the corresponding actions. The paper gives a theoretical analysis under linear dynamics assumption and shows results on four visual control tasks with several sampling-based and gradient-based planners.

Critical strengths

The proposed objective is simple and the empirical evaluation is broad. CGS is a single auxiliary loss that requires no change to the planner, and it is evaluated against LeWM and temporal straightening on four environments with both sampling-based and gradient-based planners.

Critical weaknesses
  • There is a mismatch between the theoretical guarantees and how the method is actually trained and used. The paper claims finite-budget guarantees for MPPI, contraction results for CEM, and convergence bounds for gradient descent. However, these results are derived for a linear latent model trained with a Gram-matching loss under a fixed-radius assumption, and they describe the geometry of the encoder rather than the predictor. In practice, the model they used is a nonlinear transformer, the loss is a cosine loss that ignores magnitudes, and planning uses the predictor rather than the encoder. The paper acknowledges each of these gaps in the appendix but only partially qualifies the claims in the main text.

  • The paper motivates matching action and latent-difference geometry by planner conditioning (Sec. 4), but it never states when this is a physically sensible target. Further, CGS uses action directions as the reference for shaping latent transitions. This is expected to help only when the effect of an action does not depend on the current state, that is, when the action directly determines the displacement. The paper’s own theory requires the same condition, and its own experiments show the consequence: on Reacher, where motion depends on the arm’s configuration, the gains are small and inconsistent across planners. A detailed discussion in the main text with the assumptions behind CGS and regarding the environments in which it is expected to help is needed.

Questions for authors
  • Since CGS constrains the encoder while planning uses the predictor, can the authors provide a direct quantitative analysis of whether the predictor’s predicted transitions inherit the action-aligned geometry, beyond the qualitative heatmaps of Fig. 1 / App. C.2.2 (for example, the CGS mismatch between the action and latent-difference Gram matrices computed on predicted rollouts)?

  • The hypothesis in the paper suggests that a better-conditioned planning objective should benefit both CEM and MPPI, yet the gains differ substantially and inconsistently between the two planners across environments. Can the authors explain this pattern?

4 comments

Thank you for the constructive review and for recognizing CGS's simplicity and broad evaluation. CGS's contribution is to shape action-dependent transition geometry beyond temporal straightness and connect it to finite-horizon curvature and finite-budget planning. It requires only observation–action pairs, a single auxiliary loss without reward, goal, or image-reconstruction supervision. We clarify the theoretical assumptions and provide direct evidence for predictor geometry, planner-specific effects, and the Reacher limitations below.

Q1: “Since CGS constrains the encoder while planning uses the predictor, can the authors provide a direct quantitative analysis of whether the predictor’s predicted transitions inherit the action-aligned geometry …?”

W1: “… these results are derived for a linear latent model trained with a Gram-matching loss under a fixed-radius assumption, and they describe the geometry of the encoder rather than the predictor.”

Thank you for requesting this direct test. ** Our measurements show that Predictor–Action mismatch closely tracks Encoder–Action mismatch, supporting the transfer of action-aligned geometry to the predictor.** This transfer is encouraged by the JEPA prediction loss, which trains z^t+1\hat z_{t+1} to match the encoded anchor zt+1z_{t+1}. CGS therefore shapes the geometry of the targets learned by the predictor. An encoder-side representation-shaping approach is widely used in JEPA world models, like TS (Wang et al., 2026b), Delta-JEPA (Zhang et al., 2026b), and INTACT (Sun et al., 2026), etc. Heatmap in Fig. 1 (p. 2) already visualizes predictor-based endpoint responses, and Fig. 3 (p. 6) connects predictor-derived curvature to improved MPPI updates in quadratic probes. The requested cosine-Gram mismatches directly quantify this geometric transfer below. Each cell follows LeWM / TS / CGS; lower is better.

MetricPushTCubeTwoRoomsReacher
Encoder–Action0.172 / 0.199 / 0.0290.331 / 0.304 / 0.0510.635 / 0.448 / 0.2090.295 / 0.311 / 0.212
Predictor–Action0.171 / 0.198 / 0.0340.332 / 0.306 / 0.0430.636 / 0.442 / 0.2070.295 / 0.310 / 0.253
Predictor–Encoder0.0055 / 0.0068 / 0.00390.0040 / 0.0046 / 0.00530.0023 / 0.0068 / 0.00890.0014 / 0.0012 / 0.0190

The predictor closely retains the encoder's action alignment on PushT, Cube, and TwoRooms; Reacher shows weaker transfer, showing a hard optimization problem retaining both predictive representation and CGS geometry. We will discuss more about Reacher below in the answer to W2. Our theoretical analysis inherits TS's linear-dynamics and fixed-radius assumptions and assumes shared local encoder–predictor dynamics (Appendix D.1, p. 35). The practical cosine-matching loss does not itself constrain transition magnitudes since transition and action lie in different spaces. Following your suggestion, we have clarified the connection to nonlinear dynamics through local linear approximation in the main text (Remark 4.8, pp. 6–7) and added nonlinear Gauss–Newton and full-Hessian conditioning results as further evidence that CGS shapes more balanced planning curvature of predictor rollouts to the revised manuscript (Fig. 5, p. 8; Table 13, p. 32).

Q2: “… a better-conditioned planning objective should benefit both CEM and MPPI, yet the gains differ substantially and inconsistently between the two planners across environments. Can the authors explain this pattern?”

Thank you for raising this question. Sampling-based planning depends on both the learned curvature and the proposal's coordinate-wise scales. This extends TS's gradient-based perspective and helps explain why representation improvements produce different gains across planners.

In Algorithm 1, CEM updates a vector of sampling standard deviations σp,i\boldsymbol{\sigma}_{p,i}. Let Si=diag⁡(σp,i)S_i=\operatorname{diag}(\boldsymbol{\sigma}_{p,i}). Then

\xi\sim\mathcal{N}(0,I_{n_H}),\qquad \widetilde{G}_{H,i}=S_iG_HS_i.$$ Thus, CGS shapes $G_H$, while CEM also adapts $S_i$, potentially compensating for coordinate-wise curvature imbalance. MPPI keeps $S_i$ fixed, so improvements in $G_H$ directly improve the curvature seen by sampling. This distinction is consistent with the strong LeWM CEM baseline and smaller CGS gain on PushT. The transformation is discussed in Appendix D.3.2 of the manuscript (Eqs. 66–67, p. 40); the formal planner bounds condition on an isotropic proposal.

To test this explanation, we evaluate mean-only CEM, retaining elite-based mean updates while fixing the sampling standard deviations at their initial values. We report success rates as mean ± SD (%) over three evaluation seeds, with K=128K=128 and 30 refinement steps.

PlannerModelPushTCubeTwoRoomsReacher
MPPILeWM64.0 ± 5.359.3 ± 4.692.0 ± 3.565.3 ± 4.2
MPPI+TS64.7 ± 8.375.3 ± 5.097.3 ± 3.164.0 ± 2.0
MPPI+CGS77.3 ± 6.479.3 ± 4.697.3 ± 1.266.7 ± 2.3
CEMLeWM89.3 ± 6.163.3 ± 2.376.7 ± 10.183.3 ± 8.3
CEM+TS86.7 ± 5.073.3 ± 10.397.3 ± 1.281.3 ± 8.1
CEM+CGS92.0 ± 3.576.7 ± 9.595.3 ± 1.280.7 ± 3.1
Mean-only CEMLeWM65.3 ± 2.356.7 ± 9.067.3 ± 4.661.3 ± 11.7
Mean-only CEM+TS60.7 ± 6.163.3 ± 6.494.7 ± 4.262.0 ± 5.3
Mean-only CEM+CGS75.3 ± 5.074.7 ± 7.692.0 ± 2.062.7 ± 1.2

Freezing the proposal scales increases CGS's gain over TS from 5.3 to 14.6 points on PushT and from 3.4 to 11.4 points on Cube. This supports proposal adaptation as one contributor to the smaller gains under standard CEM. Planning shows similar saturation of TS and CGS on TwoRooms, where we added geometry analysis to the revised manuscript as extra explanation (Sec. 6, Fig. 5, p. 8; Table 13, p. 32). Reacher presents a different limitation, which we discuss as below in W2.

W2: “… This is expected to help only when the effect of an action does not depend on the current state, that is, when the action directly determines the displacement. … on Reacher, where motion depends on the arm’s configuration, the gains are small and inconsistent across planners.”

Thank you for requesting clearer applicability conditions. CGS is most useful when local action relations are informative about transition geometry: it encourages similar pairwise angles without requiring actions to uniquely determine displacement or match its magnitude. The theoretical model Δzt=(A−Id)zt+Bat\Delta z_t=(A-I_d)z_t+Ba_t allows state-dependent drift; it does not assume A=IdA=I_d. Appendix D.7 (pp. 47–48) further analyzes how control geometry can be shaped when nonzero drift must be retained, and how this limits ideal straightening. We have expanded this discussion and added supporting physical probes and reference/history analyses to the revised manuscript (Secs. 5.2–6, pp. 8–10).

The physical probes in Tables 2 and 9 show that translation commands provide informative references for agent motion in PushT/TwoRooms and grasp-active cube motion. Reacher's torque-driven motion depends strongly on configuration and velocity: adding state raises fingertip-displacement prediction R2R^2 from −0.006-0.006 to 0.9830.983. Both TS and default CGS show modest, planner-dependent gains there. This motivates testing whether a different summary of the same torque inputs provides a more useful reference.

Our new experiments identify two contributors to planning failures on Reacher: difficulty learning favorable planning geometry from the default torque reference, and missing velocity information under single-frame planning. We examine these through reference and observation-history interventions, now included in Sec. 6 (pp. 9–10), Fig. 6, and Appendix C.5.5 (pp. 32–33). Default CGS-C concatenates the five primitive actions within each block. CGS-S instead sums the standardized torques, capturing cumulative actuation and cancellation. For short blocks with slowly varying torques and approximately constant local dynamics, this impulse-like quantity approximates, up to scale, the time-weighted actuation contributing to displacement (Appendix C.5.5, Eq. 28).

Changing the reference improves planning geometry, most clearly in the full nonlinear Hessian. Table 13 (p. 32) reports:

Effective condition number ↓LeWMTSCGS-CCGS-S
κeffda(GHGN)\kappa_{\mathrm{eff}}^{d_a}(G_H^{\mathrm{GN}})1.231.601.601.31
κeffda(GHfull)\kappa_{\mathrm{eff}}^{d_a}(G_H^{\mathrm{full}})11.557.668.974.12

CGS-S improves both measures relative to CGS-C and achieves the best full-Hessian conditioning. To assess how this translates into planning, we also include transition-CGS as a positive control, using privileged physical state information of whole-arm displacement as its training reference (Appendix C.5.5, Eq. 30); planning still uses images and actions.

We then examine the missing observation information in our planning protocols. Naive inputs, the default protocol, supply only the current image at every replanning call, leaving initial velocity unobserved. Online history starts from one image and retains the latest three actual observations and their two connecting executed-action blocks as memory for subsequent replanning. Privileged history additionally supplies preceding dataset observations and actions at the first solve; later solves use online memory. Each trained checkpoint remains fixed across all three protocols.

The table reports mean success (%) over three evaluation seeds, with 50 episodes each (Fig. 6, p. 9). CEM/MPPI use K=128K=128, I=30I=30; GD uses 100 steps.

PlannerHistoryLeWMTSCGS-CCGS-STransition-CGS
MPPINaive65.364.066.772.770.0
MPPIOnline69.372.768.780.079.3
MPPIPrivileged68.774.772.785.386.0
CEMNaive83.381.380.787.384.0
CEMOnline92.794.092.096.098.0
CEMPrivileged96.097.398.798.799.3
GDNaive76.781.382.081.382.7
GDOnline81.388.089.393.394.0
GDPrivileged84.089.390.794.095.3

The two interventions provide complementary benefits. Changing the reference improves naive-input MPPI and CEM success, while adding memory to the same checkpoints strengthens planning further. With online history, CGS-S exceeds TS by 7.3 points for MPPI and 5.3 points for GD. It outperforms LeWM, TS, and CGS-C under MPPI and GD in both history settings, while CEM approaches saturation. Its mean success remains close to transition-CGS across all three protocols, showing that an action-only reference can approach the privileged physical-displacement reference on this state-dependent torque task.

Returning to naive inputs, the remaining failures reveal a gap between approaching the goal and completing it precisely. Fig. 6(d) reports the fraction of failed rollouts reaching joint-angle error e<0.10e<0.10 rad; success requires e<0.05e<0.05 rad:

PlannerLeWMTSCGS-CCGS-S
MPPI79.2%78.7%76.7%82.9%
CEM84.6%85.7%89.5%95.0%
GD38.9%42.4%45.5%60.7%

Despite improved full-Hessian conditioning, CGS-S still falls short of perfect success under naive single-frame inputs. Its higher near-goal failure fraction, together with the gains from adding observation history, suggests that better geometry helps planning approach the goal, while missing velocity information limits precise completion.

These results clarify CGS's applicability: action relations must provide a useful local geometric reference, and the planner needs sufficient observation history to infer the relevant dynamics. Improving both substantially strengthens CGS on Reacher, even though torque alone does not determine displacement.

Summary of manuscript revisions

We have clarified the theoretical contribution and strengthened the empirical evidence, while reducing the full manuscript from 55 to 48 pages, including references and appendices. Sec. 4 distinguishes the framework inherited from TS from our contributions and discusses the connection between linear and nonlinear dynamics (Remark 4.8, p. 6). Table 1 (p. 8) presents the expanded cross-architecture evaluation, while Sec. 6 adds nonlinear planning-Hessian conditioning (Fig. 5, p. 8) and deeper Reacher analyses (Fig. 6, p. 9). Appendix C reorganizes the supporting protocols and analyses, alongside new alternative-regularizer comparisons (Table 14, p. 33) and an expanded CGS-weight ablation (Fig. 9, p. 25). We also refined the abstract and introduction to reflect the clarified contributions and expanded empirical evidence.

AuthorsOpenReview ↗

Follow-up before the discussion concludes

Dear Reviewer FwVA,

Thank you again for your careful review and constructive suggestions. As the discussion period draws to a close, we would like to briefly summarize our responses to your main questions:

  • Theory–practice connection: We added predictor–action and encoder–action mismatch measurements and nonlinear Hessian analyses; the results show that the predictor retains CGS’s action-aligned geometry, alongside improved conditioning of the planning objective.
  • Different gains under CEM and MPPI: CEM adapts its covariance, which can compensate for uneven planning curvature, whereas MPPI uses fixed covariance. We therefore conducted mean-only CEM experiments and found that fixing its covariance makes CGS’s gains more significant.
  • Physically meaningful action references: We tested alternative action references and planning protocols on Reacher; an action-only reference achieves planning success close to, and sometimes exceeding, that of a privileged displacement-state reference, demonstrating that action relations can provide a physically sensible target, while added memory information for planning protocols further improves performance.

We have also provided further discussion in response to your insightful comments, clarifying the theoretical assumptions, their connection to nonlinear dynamics through local linear approximation, and why CGS does not require reference (like actions) to uniquely determine displacement.

If these responses resolve your questions, we would greatly appreciate it if you could consider reflecting this in your assessment.

Best regards,
Authors

Reviewer_zu9oOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Control Geometry Straightening (CGS), a new auxiliary loss added to end-to-end latent world model training objectives. It makes the pairwise cosine similarities among latent differences match those among the corresponding actions. The paper implements the idea and shows experiment results on top of LeWM, showing that LeWM + CGS outperforms LeWM or LeWM + Temporal Straightening across four environments in the original LeWM paper.

Critical strengths
  • The paper is clearly written.
  • The method is well-motivated, and the problem of regularizing latent geometry is relevant in planning with latent world models.
Critical weaknesses
  • The method's benefit over Temporal Straightening (TS) is unclear. The performance of LeWM + TS seems on par with LeWM + CGS on some tasks, and there are no experiments on combining both TS and CGS.
  • The paper claims that the proposed regularization can be applied to other classes of world models than LeWM. However, the paper only compares the full set of experiments on top of LeWM. The paper would be stronger if the full set of experiments are also conducted on top of another class of world models, e.g. DINO-WM.
  • The paper proposes a new representation regularizer, but it only compares TS as the main baseline. The paper cited many regularizers such as inverse dynamics, SCALE, E2C, but compared with none of them.
  • Only one training seed is done for each method. The error bars came from planning seeds. Since the contribution is a change to training, the paper should report variance across several training seeds.
Questions for authors
  • How does training with TS + CGS perform? Does CGS add anything on top of TS?
  • What history N did you use for all experiments? How does CGS's performance vary with longer N?
  • How does the method compare with other representation regularization methods?
  • How sensitive is CGS with respect to its weight? How is the weight chosen for each environment? On PushT, Figure 11 has no zero-weight point and is somewhat flat from 0.01 to 0.5, so it’s unclear what the weight controls.
4 comments

Thank you for the constructive review and for recognizing the clarity, motivation, and relevance of our work. CGS shapes planning geometry through local action–transition matching, using observation–action pairs without reward, goal, or image-reconstruction supervision. Our analysis in a linear latent model connects this objective to finite-horizon curvature and finite-budget sampling-based planning. We provide the requested comparisons and clarify the ablations below.

W1: “The method's benefit over Temporal Straightening (TS) is unclear. The performance of LeWM + TS seems on par with LeWM + CGS on some tasks …”

Q1: “How does training with TS + CGS perform? Does CGS add anything on top of TS?”

Thank you for raising this point and suggesting the TS+CGS experiment. CGS's additional benefit comes from shaping action-dependent geometry beyond temporal straightness, and its effect varies across tasks.

Intuitively, TS (Wang et al., 2026b) and CGS both encourage straightening trajectories when consecutive actions have the same direction. CGS additionally preserves the angular relations between different action directions, allowing latent trajectories to turn when the actions turn. Under our stated linear-model assumptions, CGS encourages both near-identity transitions and balanced action effects (B⊤B≈cIB^\top B\approx cI), yielding more balanced finite-horizon planning curvature (Secs. 3.2–4, pp. 4–7). Temporal straightness alone does not impose this additional balance across action directions.

The gains over TS depend on how much additional action–transition structure CGS captures, with smaller differences when planning approaches saturation or is limited by dynamical ambiguity. PushT: LeWM (Maes et al., 2026) already exhibits strong implicit straightening, so TS adds little; CGS better captures action-related turns and improves planning conditioning (Figs. 1 and 5; Fig. 10, p. 26). Cube: Sustained, similarly directed actions favor TS, while CGS further improves control geometry and Gauss–Newton conditioning (Fig. 5). TwoRooms: Sampling-based planning approaches saturation, leaving limited headroom to distinguish TS and CGS. Reacher: Configuration- and velocity-dependent torque dynamics weaken the action reference, while single-frame planning lacks dynamical context. Under the default setting, gains remain modest and planner-dependent; our new analysis examines these limitations through alternative references and observation histories (Fig. 6, p. 9; Appendix C.5.5, pp. 32–33).

Following your suggestion, we additionally trained with both regularizers. CGS+TS largely matches CGS and retains its gains over TS, particularly on PushT and Cube. We report success rate (%) as mean ± SD over three evaluation seeds, with 50 episodes each; CEM/MPPI use K=128K=128, I=30I=30, and GD uses 100 steps.

PlannerMethodPushTCubeTwoRoomsReacher
CEMTS86.7 ± 5.073.3 ± 10.397.3 ± 1.281.3 ± 8.1
CEMCGS92.0 ± 3.576.7 ± 9.595.3 ± 1.280.7 ± 3.1
CEMCGS+TS91.3 ± 5.074.0 ± 6.093.3 ± 3.182.0 ± 2.0
MPPITS64.7 ± 8.375.3 ± 5.097.3 ± 3.164.0 ± 2.0
MPPICGS77.3 ± 6.479.3 ± 4.697.3 ± 1.266.7 ± 2.3
MPPICGS+TS72.0 ± 3.578.0 ± 7.298.7 ± 1.264.0 ± 8.7
GDTS82.7 ± 5.865.3 ± 4.665.3 ± 9.981.3 ± 5.0
GDCGS89.3 ± 6.471.3 ± 4.670.7 ± 3.182.0 ± 5.3
GDCGS+TS85.3 ± 4.272.7 ± 8.170.7 ± 9.580.7 ± 5.8

CGS+TS generally remains close to or below CGS, with improvements in a few settings. On TwoRooms, all three variants retain high sampling success, while Reacher results remain mixed. From our experiment, adding CGS to TS provides additional benefits, whereas adding TS to CGS offers no consistent further improvement. This is consistent with the objectives overlapping on aligned-action segments but potentially competing when actions turn.

W2: “The paper would be stronger if the full set of experiments are also conducted on top of another class of world models, e.g. DINO-WM.”

Following your suggestion, we expanded the comparison to all four environments and all three planners, using both CLS and Patch representations of DINO-WM (Zhou et al., 2025). Results are included in Table 1 (p. 8), with implementation details in Appendix C.2 (pp. 24–25) of the revised manuscript. The DINOv2 backbone (Oquab et al., 2024) remains frozen, while the adapter/projector and predictor are trainable.

CGS shows its most consistent benefits with global CLS representations, exceeding TS in mean success across all 12 settings, including TwoRooms MPPI (82.7% → 93.3%) and Cube MPPI (63.3% → 74.0%). With Patch representations, CGS is competitive with TS under sampling-based planning, while TS retains advantages in several GD settings. This pattern supports applicability beyond LeWM, with particularly consistent gains when CGS directly regularizes the global latent used for planning; patch tokens capture local visual changes, including motion across patches, which may align less directly with action geometry.

W3: “The paper cited many regularizers such as inverse dynamics, SCALE, E2C, but compared with none of them.”

Q3: “How does the method compare with other representation regularization methods?”

Thank you for this suggestion. SCALE (Hu et al., 2026) requires privileged task-relevant state during training, while E2C (Watter et al., 2015) introduces a reconstruction-based variational model with locally linear dynamics. To preserve observation–action supervision and the JEPA model class, we added inverse-dynamics and temporal-contrastive regularizers while retaining the same LeWM backbone, prediction loss, and SIGReg objective. Their definitions and results are included in Appendix C.5.6 (pp. 33–34), Table 14 (p. 33).

Inverse dynamics (LDAD). Following Delta-JEPA (Zhang et al., 2026b), a training-only decoder reconstructs actions from latent displacements:

=\frac{1}{BTd_a}\sum_{b=1}^{B}\sum_{t=0}^{T-1} \left\\|D_{\psi}(z_{b,t+1}-z_{b,t})-a_{b,t}\right\\|_2^2.$$ We test weight 10, following Delta-JEPA, and the larger value 100. Accurate action decoding preserves action information but **does not itself ensure that latent similarities reflect action similarities or that planning curvature is well conditioned**. **Temporal contrastiveness (TC).** Following the comparison in TS ([Wang et al., 2026b, Appendix B.5](https://arxiv.org/html/2603.12231v3)), we use $\mathcal{L}_{\mathrm{TC}}=\operatorname{InfoNCE}(z_{\mathrm{pos}},z_{\mathrm{neg}})$, with other frames within a temporal window of length $N$ as positives and frames outside that window in the same trajectory as negatives. This encourages temporal proximity in the representation, but **can incorrectly separate similar states revisited later in a trajectory**. We report mean PushT success rate (%) over three evaluation seeds, with 50 episodes each. CEM/MPPI use $K=128$, $I=30$; GD uses 100 steps. | Method | Weight | CEM | MPPI | GD | | --- | --- | --- | --- | --- | | LeWM | — | 89.3 | 64.0 | 85.3 | | TS | 0.01 | 86.7 | 64.7 | 82.7 | | CGS | 0.01 | **92.0** | **77.3** | **89.3** | | LDAD | 10 | 87.3 | 70.7 | 83.3 | | LDAD | 100 | 80.7 | 58.0 | 70.0 | | TC | 0.001 | 90.7 | 63.3 | 88.0 | | TC | 0.01 | 84.0 | 53.3 | 76.7 | LDAD at weight 10 improves MPPI, while TC at the smaller weight remains close to LeWM, with modest CEM and GD gains. **CGS achieves the highest mean across all three planners**, and larger weights degrade both alternatives. These results support explicitly shaping control geometry beyond action recoverability or temporal proximity, consistent with our analysis of balanced planning curvature.

W4: “Only one training seed is done for each method. The error bars came from planning seeds. … the paper should report variance across several training seeds.”

We agree that evaluation-seed variation does not address training randomness. We have now repeated LeWM, TS, and CGS with three independent training seeds: 3072, 3073, and 3074. The table reports mean ± SD across the three trained models, using each model's aggregated evaluation success rate as one observation.

PlannerMethodPushTCubeTwoRoomsReacher
CEMLeWM89.11 ± 1.0264.89 ± 0.3876.00 ± 0.6782.22 ± 3.01
CEMTS87.78 ± 3.2972.67 ± 2.3196.67 ± 0.6782.00 ± 2.91
CEMCGS92.00 ± 2.0077.33 ± 3.0696.22 ± 0.7783.78 ± 3.42
MPPILeWM67.33 ± 6.0059.11 ± 1.9291.78 ± 2.0465.11 ± 3.36
MPPITS62.44 ± 4.5474.44 ± 1.6896.89 ± 2.0465.56 ± 3.01
MPPICGS77.78 ± 2.1481.11 ± 2.0497.78 ± 0.3867.33 ± 3.06
GDLeWM84.89 ± 0.3860.89 ± 2.6939.33 ± 5.3376.22 ± 1.39
GDTS80.67 ± 3.0663.78 ± 0.3866.89 ± 0.7780.44 ± 0.77
GDCGS87.56 ± 1.5470.22 ± 1.6870.44 ± 1.3982.00 ± 1.15

CGS has a higher mean than TS in 11/12 settings, with MPPI gains of 15.3 percentage points on PushT and 6.7 points on Cube. CEM on TwoRooms remains comparable (96.22% versus 96.67%). Reacher again shows smaller gains after averaging over training seeds, consistent with the dynamical limitations discussed in Fig. 6 (p. 9). These results support the robustness of the main gains to training randomness, particularly on PushT and Cube.

Q2: “What history N did you use for all experiments? How does CGS's performance vary with longer N?”

We use N=3N=3 during training in all main experiments, including LeWM and both DINO-WM variants, retaining LeWM's default context (Sec. 3.3, p. 4; Appendix C.2, pp. 24–25). The existing PushT ablation covers N∈{2,3,4,5}N\in\{2,3,4,5\} and appears in Fig. 8, Appendix C.3 (p. 25) of the revised manuscript. CEM and GD improve slightly at N=4N=4, while MPPI performs best at N=3N=3. Thus, longer history is not uniformly better, and our gains are already obtained with LeWM's short default context.

Q4: “How sensitive is CGS with respect to its weight? How is the weight chosen for each environment? On PushT, Figure 11 has no zero-weight point and is somewhat flat from 0.01 to 0.5, so it’s unclear what the weight controls.”

The weight controls the strength of action–transition geometry matching. For LeWM, we use a common default of λCGS=0.5\lambda_{\mathrm{CGS}}=0.5 on Cube, TwoRooms, and Reacher, and a conservative value of 0.010.01 on PushT, supported by its sensitivity sweep. Each coefficient is fixed across planners and planning budgets, with a linear ramp over the first 5% of training (Secs. 3.3 and 5.1; Appendix C.1.1, p. 22).

Thank you for pointing out the missing zero-weight baseline and the unclear presentation. We now include the λCGS=0\lambda_{\mathrm{CGS}}=0 case, extend the sweep to λCGS=1\lambda_{\mathrm{CGS}}=1, and plot each planner separately in here. We also updated it to the revised manuscript as Fig. 9, Appendix C.3 (p. 25). The zero-weight point reuses the LeWM baseline checkpoint, since λCGS=0\lambda_{\mathrm{CGS}}=0 recovers its training objective. Small weights (0.010.01–0.020.02) improve all three planners over the zero-weight baseline. Larger weights generally reduce CEM and GD success, while MPPI varies non-monotonically. Separate vertical scales reveal variations that were compressed in the original shared-scale plot. This pattern is consistent with a trade-off between geometry matching and the other training objectives, supporting the conservative choice of 0.010.01 on PushT.

Summary of manuscript revisions

We have clarified the theoretical contribution and strengthened the empirical evidence, while reducing the full manuscript from 55 to 48 pages, including references and appendices. Sec. 4 distinguishes the framework inherited from TS from our contributions and discusses the connection between linear and nonlinear dynamics (Remark 4.8, p. 6). Table 1 (p. 8) presents the expanded cross-architecture evaluation, while Sec. 6 adds nonlinear planning-Hessian conditioning (Fig. 5, p. 8) and deeper Reacher analyses (Fig. 6, p. 9). Appendix C reorganizes the supporting protocols and analyses, alongside new alternative-regularizer comparisons (Table 14, p. 33) and an expanded CGS-weight ablation (Fig. 9, p. 25). We also refined the abstract and introduction to reflect the clarified contributions and expanded empirical evidence.

AuthorsOpenReview ↗

Follow-up before the discussion concludes

Dear Reviewer zu9o,

Thank you again for your careful review and constructive suggestions. As the discussion period draws to a close, we would like to briefly summarize our responses to your questions:

  • Additional benefit over TS: We conducted new experiments combining TS and CGS; the results show that adding CGS improves upon TS, demonstrating the value of shaping action-dependent geometry beyond temporal straightness.
  • Applicability beyond LeWM: We expanded the DINO-WM experiments to all four environments and three planners, with both CLS and Patch representations; CGS exceeds TS in mean success across all 12 CLS settings, demonstrating its benefits across multiple world model architectures.
  • Advantages over other regularizers: We conducted new comparisons with inverse-dynamics and temporal-contrastive regularizers under the same backbone and supervision; CGS achieves the highest mean success across all three planners in these comparisons, highlighting the benefit of shaping control geometry beyond inverse dynamics and contrastive regularization.

We have also provided further analyses and discussion in response to your insightful comments, including robustness across three independent training seeds, the effect of history length, and the role and selection of the regularization weight.

If these responses resolve your questions, we would greatly appreciate it if you could consider reflecting this in your assessment.

Best regards,
Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank you for your time, thoughtful feedback, and constructive suggestions.

We sincerely thank the reviewers for their careful reviews, thoughtful feedback, and constructive suggestions. We summarize our rebuttal and revisions as follows:

Reviewer zu9o: We thank the reviewer for the feedback. The reviewer finds the paper clearly written, well-motivated, and relevant to latent-world-model planning. The questions mainly concern CGS’s benefit over TS and the breadth and robustness of the evidence. We conducted new experiments combining TS and CGS, comparing alternative regularizers, repeating training across three seeds, and extending DINO-WM evaluation, strengthening the evidence for CGS’s additional benefits and generality.

Reviewer FwVA: We thank the reviewer for recognizing the simple objective and broad evaluation. The questions concern the theory–practice connection, varying gains across planners, and the action reference. We clarified the assumptions and provided new results supporting the transfer of action-aligned geometry to the predictor. We also conducted new experiments with mean-only CEM to explain the varying gains across planners, and with alternative action references on Reacher to clarify when action geometry provides a physically meaningful reference.

Reviewer AZi5: We are glad the reviewer finds that actions provide useful information beyond temporal straightening and appreciates the method’s simplicity. The questions mainly concern when action similarity is informative and when CGS improves over TS. We provided new experiments on CGS references and planning protocols on Reacher, showing complementary benefits from improved control geometry and velocity information available through the planner’s memory. We also clarified the theoretical advances over TS in the main text and grouped the main results by planner.

Reviewer F2HG: We thank the reviewer for recognizing our simple approach to an important problem in world-model planning, the rigorous and informative theoretical analysis, and the breadth of our evaluation. The reviewer raised questions about the rationale behind CGS, the theory–practice gap in nonlinear dynamics, CGS’s applicability, and broader empirical validation. We explained CGS’s design rationale, discussed when a local linear interpretation is appropriate, and provided new results on nonlinear full-Hessian conditioning and comparisons with alternative regularizers. We also added new experiments for a deeper analysis of Reacher, showing that effective planning benefits from both an improved planning Hessian and sufficient velocity information through the planner’s memory. Expanded cross-architecture results with DINO-WM are now included in the main text.

Following the reviewers’ suggestions, we condensed the manuscript from 55 to 48 pages, while adding more results and discussion they suggested. Thank you again for all the constructive suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.