acceptodds
Under review as a conference paper at ICLR 2027

Multiple Shooting for Gradient-Based Planning with World Models

Abstract

Planning with world models is typically framed as a trajectory optimization problem where the goal is to find an action sequence that drives a system from an initial to a target state. Since world models are trained purely for prediction, the task-specific objective is specified only at inference, making the planner a key determinant of success. Standard approaches formulate this as a single shooting problem, where actions are the only optimization variables and the objective is evaluated by unrolling a learned latent dynamics model. However, optimizing through these rollouts compounds model errors, exposes the optimizer to many local minima, and is inherently sequential. Multiple shooting formulations sidestep these issues by treating states as additional optimization variables but traditionally rely on second-order methods that scale poorly to high-dimensional latent spaces. We propose PALM, a first-order multiple shooting planner for learned latent spaces. PALM treats latent states as explicit optimization variables and enforces dynamics consistency through an augmented Lagrangian. By relaxing dynamics constraints during optimization, PALM can move through infeasible regions and escape local minima where single shooting methods typically get stuck. We evaluate PALM on navigation (Wall, PointMaze) and manipulation (PushT) tasks. On Wall and PointMaze, PALM outperforms gradient-based baselines and degrades more slowly as the goal gets further: on the longest Wall crossing tasks, it succeeds on of tasks, against at most for other gradient-based planners. CEM achieves higher final success, but PALM reaches competitive success in less wall-clock time. On PushT, all planners reach similar success rates.

Primary area:
applications to robotics, autonomy, planning
Keywords:
planning, optimization, world models

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_6CxEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PALM, a first-order augmented Lagrangian planner for learned latent dynamics. The method jointly optimizes intermediate latent states and actions, enforcing dynamics consistency through vector-valued multipliers and adaptive penalty coefficients, while using Adam to approximately solve the inner subproblems. Using the same world model, the experiments compare PALM with single-shooting, multiple-shooting, and sampling-based planners. The results show that PALM substantially outperforms the evaluated gradient-based planners on several challenging navigation tasks and outperforms CEM under certain computational budgets.

Critical strengths
  1. The paper presents a well-motivated planning approach with clear methodological distinctions. PALM combines first-order optimization with an augmented Lagrangian treatment of vector-valued dynamics residuals, distinguishing it from the second-order LatCo implementation evaluated in this paper and the pure-penalty approach used by GRASP. Although the underlying optimization tools are established, their effective adaptation to high-dimensional learned latent spaces has both research and practical value. The paper also clearly positions the approach relative to prior work.
  2. The experiments demonstrate practically meaningful performance improvements. All planners share the same world model, planning instances, and horizons, and LatCo is adapted to the same 1,578-dimensional latent state. On the longest Wall crossing tasks, PALM achieves an 82.3% success rate, compared with 31.0% for SS-Grad and 22.7% for GRASP. The success-versus-time curves further demonstrate its value under limited planning budgets.
Critical weaknesses

The individual contributions of the key design choices have not been established through controlled ablations. The comparisons with LatCo and GRASP support the effectiveness of PALM as a complete approach. However, these methods differ simultaneously in their constraint formulations, optimizers, gradient handling, and update settings, making it difficult to determine how much improvement comes from multiplier updates and adaptive penalties, respectively. Since augmented Lagrangian constraint handling is an important aspect of the proposed method, comparing the full PALM with variants that disable multiplier updates or use fixed penalties, while keeping other settings unchanged, would help identify the sources of the performance gains.

Questions for authors
  1. Which key design choices account for PALM’s performance gains? Could you compare the full PALM with variants that disable multiplier updates or use fixed penalty coefficients, while keeping other settings unchanged? Reporting success rates and dynamics residuals under the same computational budget would help clarify the individual contributions of multiplier updates and adaptive penalties.
  2. How do the authors interpret the trade-off between computational efficiency and success rate when comparing PALM with CEM? The paper demonstrates PALM’s advantages under certain time budgets and CEM’s higher final success rates on some tasks. What task characteristics or computational conditions favor choosing PALM?
  3. What empirical observations do the authors have regarding PALM’s dependence on world-model accuracy? The paper discusses model error through its tolerance settings and a failure case. Building on this analysis, have the authors observed whether PALM is more or less prone to exploiting model errors than other planners? Such a discussion would help clarify the conditions under which the method is effective.
  4. Which design choices can be carried over directly to broader planning settings, and which might require modification? For example, would the current initialization, constraint-update rules, and stopping criteria remain suitable for closed-loop MPC or tasks involving intermediate-state costs?
2 comments
AuthorsOpenReview ↗

Answer to reviewer 6CxE 1/2

We thank the reviewer for their positive review, and answer their questions and concerns below.

The individual contributions of the key design choices ... would help identify the sources of the performance gains.

Thank you for raising this point! We agree, and we added an ablation of PALM (see our answer to the first question below).

Which key design choices account for PALM’s performance gains? Could you compare the full PALM with variants that disable multiplier updates or use fixed penalty coefficients, while keeping other settings unchanged? Reporting success rates and dynamics residuals under the same computational budget would help clarify the individual contributions of multiplier updates and adaptive penalties.

We added these ablations in Table T1 (comment “To all reviewers”). We compare PALM with variants that remove the multiplier updates, fix the penalty coefficients, or both, under the same computational budget and report SR and dynamics residuals.

How do the authors interpret the trade-off between computational efficiency and success rate when comparing PALM with CEM? The paper demonstrates PALM’s advantages under certain time budgets and CEM’s higher final success rates on some tasks. What task characteristics or computational conditions favor choosing PALM?

We believe that because CEM samples a pool of candidates to evaluate at every iteration, given enough time it is able to explore the non-convex space better and reach higher final success. PALM on the other hand optimizes a single candidate (actions and nodes), which makes iterating faster and allows it to reach competitive success faster. We expect PALM to be preferable in two settings:

  • Larger action spaces, where the higher action dimension makes the space CEM samples from larger. To support this, we added experiments on a new environment, Cube, with 5-dimensional actions instead of 2 (Table T2, comment “To all reviewers”), and found that PALM reaches higher SR than CEM at all goal distances, with a planning time 5x lower.
  • Closed-loop MPC. In this case, because we only execute the first actions of a plan, a good plan obtained quickly matters more than the best plan obtained slowly. We added closed-loop MPC experiments with PALM and SS-Grad in Table T6. Due to limited compute, we will add MPC experiments with sampling-based planners in the final version.

What empirical observations do the authors have regarding PALM’s dependence on world-model accuracy? The paper discusses model error through its tolerance settings and a failure case. Building on this analysis, have the authors observed whether PALM is more or less prone to exploiting model errors than other planners? Such a discussion would help clarify the conditions under which the method is effective.

We measured how planners exploit model errors by comparing the latent goal distance reached by their final actions in the world model and in the environment (Table T5). Sampling-based planners exploit model errors least on Wall and PointMaze, consistent with the observation that gradient-based optimization is more prone to finding actions where the model is inaccurate [1]. Among gradient-based planners, PALM has the smallest gap on all tasks except PointMaze passthrough, and the smallest on average. One explanation is that PALM calls the model only one step ahead and sets tolerance floors based on the model prediction error, preventing optimization beyond model accuracy.
PALM depends on the one-step accuracy of the model: it stops when the residuals reach the model’s one-step error, so small errors at each step can add up over the rollout. This is the case on PointMaze, where the forward rollout drifts from the shooting nodes (Table T4, comment to reviewer RG5x). PALM is therefore most effective when the one-step prediction error of the model is small. In closed-loop MPC, replanning corrects this drift (Table T6).

Table T5. Gap between the latent goal distance reached in the environment and in the world model (×10⁻³, lower is better). Median over episodes. Bold: best among gradient-based planners.

PlannerWall sameWall crossingPointMaze passthroughPointMaze restPushTAll tasks
CEM1.14 ± 0.012.90 ± 0.1130.2 ± 0.842.0 ± 1.325.2 ± 0.520.3 ± 0.4
MPPI0.339 ± 0.0190.568 ± 0.03017.4 ± 0.65.74 ± 0.2033.7 ± 1.111.5 ± 0.3
SS-Grad2.07 ± 0.096.47 ± 0.1140.8 ± 1.6106 ± 524.2 ± 0.435.9 ± 0.9
GRASP7.03 ± 0.0827.2 ± 0.536.4 ± 0.9107 ± 0.428.6 ± 0.441.3 ± 0.1
PALM1.90 ± 0.195.32 ± 0.2139.1 ± 1.372.3 ± 5.118.0 ± 0.227.3 ± 0.9

[1] Parthasarathy et al., Closing the Train-Test Gap in World Models for Gradient-Based Planning, arXiv, 2025.

AuthorsOpenReview ↗

Answer to reviewer 6CxE 2/2

Which design choices can be carried over directly to broader planning settings, and which might require modification? For example, would the current initialization, constraint-update rules, and stopping criteria remain suitable for closed-loop MPC or tasks involving intermediate-state costs?

  • In principle, all design choices can be carried over to closed-loop MPC. We added closed-loop MPC experiments on PointMaze rest goals (Table T6) with the constraint-update rules and stopping criteria unchanged. We used the same initialization for the first planning step, and at each replanning step, we warm-started the nodes and actions from the previous solution, shifted by one step. We see that with MPC, PALM’s SR increases at all goal distances. We also ran MPC with SS-Grad, to show that PALM's performance does not come from replanning alone. Due to limited compute, we ran these experiments on PointMaze rest goals only. We picked this task because PALM performs worst there in open loop relative to sampling-based planners, and its SR drops the most as the goal gets further. We will include MPC results on all environments in the final version.

Table T6. SR (%) on PointMaze rest goals, open loop (T = k/5) vs. closed-loop MPC (T = 5, 15 replanning steps). 50 episodes per k.

Plannerk = 25k = 30k = 35k = 40
Open loop
SS-Grad51.3 ± 8.430.7 ± 0.920.0 ± 5.721.3 ± 4.1
PALM74.7 ± 4.158.0 ± 5.955.3 ± 1.940.0 ± 7.1
MPC, warm start
SS-Grad54.0 ± 5.947.3 ± 3.448.0 ± 5.746.0 ± 1.6
PALM89.3 ± 3.480.7 ± 5.274.7 ± 5.770.0 ± 3.3
  • Intermediate-state costs fit the multiple shooting formulation directly: the costs ∑tℓ(zt)\sum_t \ell(z_t) would be added to the objective of Eq. 2, and the constraint update rules and stopping criteria stay the same. What might require tuning is the weight of these costs relative to the penalty coefficients ρt\rho_t.

We hope these answers address the reviewer's concerns, and we are happy to answer any further questions during the discussion period.

Reviewer_vMc1OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the problem of planning with learned, differentiable models. They point out that although single shooting methods like CEM are common, multiple shooting methods could be better suited for gradient based planning. They show superior performance to other gradient based baselines and similar final performance to CEM, while needing lesser wall clock time.

Critical strengths
  • The paper presents a technically sound approach in a clear presentation.
  • They make a good point for why multiple shooting might be a better fit for learned models than CEM type methods that are common in the JEPA literature. The fact that you can use one step errors even in the planning process sticks to the trained model's strengths, which seems like a sensible strategy for gradient based planning on top of learned models.
  • I found the empirical evaluation strong, and I appreciated that there were no over-claims and the paper was generally forthright in its evals and writing.
Critical weaknesses
  • The paper could do a better job at discussing the pros and cons of gradient vs sampling based approaches to planning with learned models.
  • In that vein, I felt like the authors should have included domains which demonstrate the effectiveness of gradient based planning. For instance, its possible that gradient methods scale better with larger action and latent spaces and longer horizons. Yet the environments in the paper were trivially small. Which likely also explains why CEM dominates in success rate, but likely wont if you scale the problem size.
  • The wall clock superiority against CEM seems insufficient. Why not compare against MPPI or other more sophisticated methods? Vanilla CEM seems like a straw-man for a speed claim.
  • This is maybe more of a complaint against most JEPA papers: but could you cite papers from before Planet and Dreamer? Off the top of my head, papers like Levine's guided policy search, embed to control, or even PILCO are classic model-based RL approaches.
Questions for authors

Could you try these comparisons in a larger env to demonstrate the capability of gradient based planning? Also comparing to MPPI would be a nice addition, although not crucial if you add a disclaimer in the wall-clock time results discussion.

4 comments
AuthorsOpenReview ↗

Answer to reviewer vMc1 1/2

We thank the reviewer for their positive review, and answer their questions and concerns below.

The paper could do a better job at discussing the pros and cons of gradient vs sampling based approaches to planning with learned models.

We agree and added a discussion of the pros and cons of both approaches in Sec. 4.6 of the revised paper. We summarize it below:

We find that sampling-based planners explore the non-convex objective more broadly and exploit model errors less on Wall and PointMaze (Table T5, comment to reviewer 6CxE). They reach the highest final SR on these tasks (Table T7), with the largest gap on PointMaze rest goals. Their cost is the number of samples: each iteration requires one full rollout per sample. Gradient-based planners on the other hand have cheaper iterations: one backward pass gives a descent direction in all action dimensions, whereas the number of samples a sampling-based planner needs grows with the action dimension. Our experiments on Cube, which has a 5-D action space, show that PALM reaches higher SR than CEM and MPPI (Table T2, comment "To all reviewers"). Gradient-based planners are local and follow the gradients of the model, so they can converge to poor stationary points (Table T3, comment to reviewer RG5x) and exploit model errors more (Table T5, comment to reviewer 6CxE). PALM reduces both effects but does not remove them. PALM reaches 50% SR much faster than sampling-based planners (Table T8). We believe this is important, especially in closed-loop MPC, which is a common setting in robotics: each replanning step is a full solve, so the lower cost of PALM’s iterations directly reduces the cost of replanning. We will add sampling-based planners to the MPC experiments (Table T6, comment to reviewer 6CxE) in the final version.

In that vein, I felt like the authors should have included domains which demonstrate the effectiveness of gradient based planning. For instance, it's possible that gradient methods scale better with larger action and latent spaces and longer horizons. Yet the environments in the paper were trivially small. Which likely also explains why CEM dominates in success rate, but likely won't if you scale the problem size.

We added experiments on the Cube environment, with 5-D actions instead of 2-D (Table T2, comment “To all reviewers”). On Cube, PALM reaches higher SR than CEM and MPPI with a planning time 5x lower.

The wall clock superiority against CEM seems insufficient. Why not compare against MPPI or other more sophisticated methods? Vanilla CEM seems like a straw-man for a speed claim.

We agree that MPPI is a stronger sampling-based baseline. We ran MPPI on the same tasks (Table T7), and report runtimes on the tasks of Fig. 4 (Wall crossing, k = 50 and PointMaze passthrough, k = 25) in Table T8. Table T7 shows that on average, MPPI results are on par with CEM results, but the runtimes show that MPPI reaches higher SR faster than CEM does, while PALM still reaches 50% SR faster than both.

Table T7. SR (%) ↑ on the 25 evaluation sets of paper Tab. 1.

PlannerWall sameWall crossingPointMaze passthroughPointMaze restPushTAll tasks
CEM99.8 ± 0.197.4 ± 0.191.7 ± 0.486.8 ± 0.439.7 ± 0.583.1 ± 0.2
MPPI99.3 ± 0.196.0 ± 0.590.2 ± 1.096.8 ± 0.630.4 ± 0.282.5 ± 0.3
PALM98.6 ± 0.385.6 ± 0.187.9 ± 0.860.6 ± 0.537.6 ± 0.374.1 ± 0.1

Table T8. Planning time (s) ↓ on the tasks of Fig. 4 of the paper, batch of 50 start–goal pairs on one A100.

PlannerWall crossing, k = 50PointMaze passthrough, k = 25
Total time (s)Time to 50% SR (s)Time to 70% SR (s)Total time (s)Time to 50% SR (s)Time to 70% SR (s)
CEM65026.329.532520.830.7
MPPI65119.724.132612.113.7
PALM15714.424.5412.58.5

This is maybe more of a complaint against most JEPA papers: but could you cite papers from before Planet and Dreamer? Off the top of my head, papers like Levine's guided policy search, embed to control, or even PILCO are classic model-based RL approaches.

We agree that prior works on model-based RL approaches should be cited. We added a paragraph on these methods in the related work section of our revised paper, and will also copy it in a second comment below for convenience.

Could you try these comparisons in a larger env to demonstrate the capability of gradient based planning? Also comparing to MPPI would be a nice addition, although not crucial if you add a disclaimer in the wall-clock time results discussion.

Please see our answers above (Cube: Table T2; MPPI: Tables T7 and T8).

We hope these answers address the reviewer's concerns, and we are happy to answer any further questions during the discussion period.

AuthorsOpenReview ↗

Answer to reviewer vMc1 2/2

We include the paragraph we added to the related work section of our revised paper for convenience:

Model-based RL methods learn dynamics models and use them for policy search or trajectory optimization, such as PILCO [1], guided policy search [2], and MPC with neural network dynamics [3]. From pixel observations, Embed to Control (E2C) [4] learns locally linear latent dynamics and uses iLQR to plan in the learned latent space. Other methods learn dynamics from pixels and plan using LQR [5], trajectory optimization [6], or sampling-based planning [7]. These methods either learn a policy, restrict the model (locally linear dynamics, low-dimensional latents), or plan using single shooting. In contrast, PALM can be applied at inference with any differentiable predictor with high-dimensional latents, and treats latent states as optimization variables.

[1] Marc Peter Deisenroth and Carl Edward Rasmussen. PILCO: A model-based and data-efficient approach to policy search. ICML, 2011.
[2] Sergey Levine and Vladlen Koltun. Guided policy search. ICML, 2013.
[3] Anusha Nagabandi, Gregory Kahn, Ronald S. Fearing, and Sergey Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. ICRA, 2018.
[4] Manuel Watter, Jost Tobias Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. NeurIPS, 2015.
[5] Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew J. Johnson, and Sergey Levine. SOLAR: Deep structured representations for model-based reinforcement learning. ICML, 2019.
[6] Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, and Pieter Abbeel. Deep spatial autoencoders for visuomotor learning. ICRA, 2016.
[7] Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. ICRA, 2017.

Thank you for the additional work and results. I have increased my score to a 4 specially in light of the higher dim experiments. Hope you also carry that related work discussion to the camera-ready.

Thank you for raising your score! We commit to keeping the new related work paragraph in the final version (we already included it in the revised version).

Reviewer_RG5xOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PALM, a first-order multiple shooting planner for learned world models. It jointly optimizes actions and latent states, using an augmented Lagrangian to reconcile temporarily inconsistent states with the learned dynamics. The formulation draws on classical constrained optimization rather than a new convergence theorem. Experiments compare planners on navigation and manipulation, examine optimization trajectories and residuals, and measure success against planning time. The reported benefits are strongest on navigation tasks requiring detours. CEM can achieve higher final success, while manipulation results show little advantage for PALM.

Critical strengths
  1. The paper clearly develops the connection between trajectory optimization and learned latent dynamics. Eqs. 2 and 3 make the role of intermediate states, dynamics constraints and multipliers easy to follow.

  2. The mechanism is well motivated. Separating shooting nodes from forward rollouts allows the optimizer to search beyond the trajectories reachable at each intermediate iteration, while the local dynamics constraints permit parallel evaluation without dense second-order solves.

  3. The evaluation in Tab. 1 and Fig. 1 uses shared world models and planning instances across methods. Varying expert path length tests a meaningful difficulty axis rather than relying on elapsed time along arbitrary dataset trajectories.

  4. Figs. 2 and 3 connect the algorithm to observable optimization behavior by comparing shooting nodes, learned-model rollouts and environment execution. This is more informative than reporting task success alone.

  5. The empirical account is appropriately qualified. Fig. 4 examines the success versus time tradeoff, while the discussion reports the stronger final CEM results, the limited manipulation benefit and the model-exploitation failure illustrated in Fig. 9.

Critical weaknesses
  1. SS-Grad in Eq. 1 uses a nonzero action penalty, whereas PALM uses none, as specified in Tabs. 3 and 4. Their optimization budgets also differ, and endpoint loss alone does not establish stationarity. The comparison therefore does not establish that dynamics relaxation avoids a worse local minimum of the same objective.

  2. The comparison with GRASP in Tab. 1 changes the objective, optimization settings and auxiliary mechanisms together. Without a matched multiplier-free PALM control, it does not isolate the contribution of multiplier updates from the other design choices. The timing curves in Fig. 4 support the reported planner tradeoffs, but do not resolve this component attribution.

Questions for authors
  1. For the stopping rule in Sec. 3.3 and step 26 of Alg. 1, what are the final per-instance maximum dynamics residuals and the frequencies of each termination condition? The tolerance floors and finite inner solves make this an approximate feasibility criterion. Fig. 2 shows mean residual trends at a short horizon, so it does not establish how closely shooting nodes and forward rollouts agree in the hardest tasks.

  2. How is the history-dependent predictor described in App. A.2.1 incorporated into the constraints of Eq. 2 and the residual computation in step 9 of Alg. 1?

6 comments
AuthorsOpenReview ↗

Answer to reviewer RG5x 1/2

We thank the reviewer for their detailed review and answer their questions and concerns below.

  • SS-Grad in Eq. 1 uses a nonzero action penalty, whereas PALM uses none, as specified in Tabs. 3 and 4. Their optimization budgets also differ, and endpoint loss alone does not establish stationarity. The comparison therefore does not establish that dynamics relaxation avoids a worse local minimum of the same objective.

We deliberately used a nonzero action penalty for SS-Grad: we tuned αu\alpha_u and kept the value that gave the best SR. Let us denote J(u)=∥f[T](z0,u)−zg∥22+αuTd∑t=0T−1∥ut∥22J(u) = \lVert f^{[T]}(z_0, u) - z_g \rVert_2^2 + \frac{\alpha_u}{Td} \sum_{t=0}^{T-1} \lVert u_t \rVert_2^2 the SS-Grad objective. We also ran SS-Grad (i.e., minimization of J(u)J(u)) with αu=0\alpha_u=0 and 40004000 Adam steps (same αu\alpha_u and budget as PALM) and report success rates (SR) in Table T3. SR drops by 5 to 6 points on all three tasks, so the gap with PALM on Wall and PointMaze does not come from the optimization budget or action penalty. Table T3 also reports the final objective JJ and the gradient norm ∥∇uJ∥\lVert \nabla_u J \rVert for both SS-Grad and PALM. With matched objective and budget, SS-Grad reaches gradient norms up to 6x smaller than PALM on Wall and PointMaze, while its objective is up to 4.5x higher, which indicates that SS-Grad reaches worse stationary points. PALM’s gradient norm is not expected to vanish since PALM does not minimize JJ directly but instead solves the constrained problem of Eq. 2, and stops once residuals reach the model accuracy. We added this table and discussion to App. A.5.2 (Tab. 9) of the revised paper, and refer to it in Sec. 4.4.

Table T3. SR, final objective JJ and gradient norm (median over episodes) averaged over the sets of paper Tab. 1.

TaskPlannerSR (%) ↑JJ (αu=0\alpha_u = 0) (×10⁻²) ↓∥∇uJ∥\lVert \nabla_u J \rVert (αu=0\alpha_u = 0) (×10⁻³)
WallSS-Grad (Tab. 1, αu=0.01\alpha_u = 0.01)72.2 ± 0.517.3 ± 0.72.94 ± 0.06
SS-Grad (αu=0\alpha_u = 0, 4,000 steps)66.3 ± 0.511.9 ± 0.41.74 ± 0.02
PALM92.1 ± 0.13.59 ± 0.0910.6 ± 0.2
PointMazeSS-Grad (Tab. 1, αu=0.1\alpha_u = 0.1)42.4 ± 0.826.6 ± 0.428.2 ± 0.4
SS-Grad (αu=0\alpha_u = 0, 4,000 steps)36.0 ± 0.227.3 ± 0.55.13 ± 0.05
PALM74.3 ± 0.36.07 ± 0.2327.6 ± 0.3
PushTSS-Grad (Tab. 1, αu=0.01\alpha_u = 0.01)40.3 ± 0.42.55 ± 0.014.07 ± 0.01
SS-Grad (αu=0\alpha_u = 0, 4,000 steps)35.1 ± 0.23.26 ± 0.052.28 ± 0.02
PALM37.6 ± 0.33.94 ± 0.0422.3 ± 0.5

The comparison with GRASP in Tab. 1 changes the objective, optimization settings and auxiliary mechanisms together. Without a matched multiplier-free PALM control, it does not isolate the contribution of multiplier updates from the other design choices. The timing curves in Fig. 4 support the reported planner tradeoffs, but do not resolve this component attribution.

Thank you for raising this point! We agree and we added ablations in Table T1 (comment “To all reviewers”). We compare PALM with variants that remove the multiplier updates, fix the penalty coefficients, or both, under the same computational budget. The matched multiplier-free PALM control is “PALM w/o multipliers”, which removes the multiplier updates and keeps all other settings unchanged. We added Table T1 to App. A.3 (Tab. 5) of the revised paper and summarize it in Sec. 4.7.

AuthorsOpenReview ↗

Answer to reviewer RG5x 2/2

For the stopping rule in Sec. 3.3 and step 26 of Alg. 1, what are the final per-instance maximum dynamics residuals and the frequencies of each termination condition? The tolerance floors and finite inner solves make this an approximate feasibility criterion. Fig. 2 shows mean residual trends at a short horizon, so it does not establish how closely shooting nodes and forward rollouts agree in the hardest tasks.

Table T4 reports the final per-instance maximum dynamics residuals per modality, the frequencies of each termination condition, and the maximum gap between the shooting nodes and the forward rollout of the planned actions, relative to the start–goal distance.
PALM stops when (i) all residuals fall below τconv\tau_{\text{conv}}, (ii) all intermediate tolerances reach their floors τmin⁡m\tau^m_{\min}, or (iii) after Kmax⁡=800K_{\max} = 800 outer iterations. We agree that this is an approximate feasibility criterion, but because we set τmin⁡m\tau^m_{\min} based on the one-step prediction error of the world model, PALM stops when the plan is consistent with the dynamics, up to the accuracy of the model, which is the best precision the model allows.
Table T4 also shows that at the end of planning, the dynamics residuals are close to the floor on Wall and PointMaze, including on the hardest sets. On Wall, the forward rollout stays close to the shooting nodes. On PointMaze, small errors at each step add up, and the rollout drifts from the nodes. On PushT, residuals stay higher, and PALM often stops at the iteration limit.

Table T4. Final dynamics residuals, frequency of each stopping condition, and maximum node–rollout gap for PALM.

SR (%) ↑Max res. / τmin⁡\tau_{\min}, visual ↓Max res. / τmin⁡\tau_{\min}, proprio ↓(i) Res. below τconv\tau_{\text{conv}} (%)(ii) Tol. at floor (%)(iii) Iteration limit (%)Max node–rollout gap ↓
Average over sets
Wall92.1 ± 0.10.54 ± 0.000.98 ± 0.0121.6 ± 0.172.7 ± 0.15.7 ± 0.20.033 ± 0.000
PointMaze74.3 ± 0.31.17 ± 0.010.56 ± 0.010.0 ± 0.094.1 ± 0.25.9 ± 0.20.228 ± 0.001
PushT37.6 ± 0.30.80 ± 0.013.70 ± 0.340.0 ± 0.051.8 ± 0.748.2 ± 0.70.152 ± 0.001
Hardest set
Wall crossing, k = 5078.1 ± 0.70.62 ± 0.011.03 ± 0.0517.8 ± 0.764.9 ± 1.417.3 ± 1.40.037 ± 0.001
PointMaze rest, k = 4041.9 ± 3.01.29 ± 0.020.69 ± 0.010.0 ± 0.082.4 ± 0.217.6 ± 0.20.340 ± 0.001
PushT 100 px, 1.5 rad11.6 ± 0.71.46 ± 0.0110.8 ± 0.30.0 ± 0.033.1 ± 0.666.9 ± 0.60.157 ± 0.002

How is the history-dependent predictor described in App. A.2.1 incorporated into the constraints of Eq. 2 and the residual computation in step 9 of Alg. 1?

In all experiments, we use the predictor with a history H=1H=1, so each constraint of Eq. 2 (in the paper) directly uses f(zt,ut)f(z_t, u_t). The predictor is trained with H=3H=3, but because it is trained with causal masking, it also learns to predict from a single latent at the first temporal position.
Our formulation extends directly to the general case with HH latents. Let it=max⁡(0,t−H+1)i_t = \max(0, t - H + 1). The dynamics constraints become ct=zt+1−f(zit:t,uit:t),t=0,…,T−2,c_t = z_{t+1} - f(z_{i_t:t}, u_{i_t:t}), \quad t = 0, \dots, T-2, and the goal term becomes ∥zg−f(ziT−1:T−1,uiT−1:T−1)∥22\lVert z_g - f(z_{i_{T-1}:T-1}, u_{i_{T-1}:T-1}) \rVert_2^2. Each constraint still calls ff only once, and all constraints can still be evaluated in parallel. Step 9 of Alg. 1 changes only in the arguments of ff, and the rest of the algorithm is unchanged. We added this to App. A.2.1.

We hope these answers address your concerns, and we are happy to answer any further questions during the discussion period.

Thank you for the additional controls and diagnostics.

While Tab. 9 strengthens the comparison, it does not establish that the worse SS-Grad solutions in Fig. 7 are local minima. This was the concern in my W1. The authors need to revise this interpretation.

Comments on the paper

2 comments
AuthorsOpenReview ↗

To all reviewers

We thank the reviewers for the time spent on our paper and the suggestions they made to improve it. We are glad that they found the approach well motivated, the presentation clear, the evaluation strong and the claims appropriately qualified. To answer their questions and concerns, we ran the following new experiments:

  • Ablations of PALM’s components (T1, reviewers RG5x and 6CxE): Removing the multipliers drops SR by 6 points and more than doubles the rollout loss. Fixed penalty coefficients reach similar SR only when tuned carefully.
  • Experiments on a new environment: Cube (T2; vMc1, 6CxE). PALM reaches higher SR than CEM and MPPI, with a planning time 5x lower.
  • SS-Grad with matched objective and budget (T3, reviewer RG5x). SR drops by 5 to 6 points, and SS-Grad reaches worse stationary points than PALM on Wall and PointMaze.
  • Final dynamics residuals and stopping-criterion statistics (T4, reviewer RG5x). On Wall and PointMaze, final residuals are at most 1.3x the tolerance floor. On PointMaze, the forward rollout drifts from the shooting nodes.
  • Model exploitation (T5, reviewer 6CxE). Among gradient-based planners, PALM exploits model errors the least on average.
  • Closed-loop MPC (T6, reviewer 6CxE). With MPC, PALM’s SR increases and stays above SS-Grad with MPC.
  • MPPI baseline (T7, T8, reviewer vMc1). MPPI reaches SR on par with CEM on average, and PALM still reaches 50% SR faster than both.

Unless stated otherwise, we report mean ± std over 3 seeds. Due to limited compute, we ran closed-loop MPC on one task (PointMaze rest goals) with PALM and SS-Grad, and we ran the experiments on Cube with PALM, CEM and MPPI. We will add Cube and the closed-loop MPC experiments to the main text of the final version, once we have run all baselines on Cube, and MPC on all environments and with sampling-based planners. We will also add the runtimes of the ablation variants.

Ablations

PALM optimizes the augmented Lagrangian L(u,z,λ)=αg2∥zg−f(zT−1,uT−1)∥22+∑t=0T−2λt⊤ct+∑t=0T−2ρt2∥ct∥22,\mathcal{L}(u, z, \lambda) = \frac{\alpha_g}{2} \lVert z_g - f(z_{T-1}, u_{T-1}) \rVert_2^2 + \sum_{t=0}^{T-2} \lambda_t^\top c_t + \sum_{t=0}^{T-2} \frac{\rho_t}{2} \lVert c_t \rVert_2^2, with residuals ct=zt+1−f(zt,ut)c_t = z_{t+1} - f(z_t, u_t), multipliers λt\lambda_t, and penalty coefficients ρt\rho_t.

Table T1. Ablation of PALM components on the evaluation sets of paper Tab. 1. All variants use the same budget of at most Kmax⁡=800K_{\max} = 800 outer iterations × K=5K = 5 Adam steps. Dynamics residual: 1T−1∑t=0T−2∥ct∥22\frac{1}{T-1} \sum_{t=0}^{T-2} \lVert c_t \rVert_2^2. Rollout loss: ∥f[T](z0,u)−zg∥22\lVert f^{[T]}(z_0, u) - z_g \rVert_2^2.

VariantMultipliers λ\lambdaρ\rhoConstraintSR (%) ↑Dynamics residual (×10⁻³) ↓Rollout loss (×10⁻²) ↓
PALM w/o multipliers✗adaptiveper dimension68.2 ± 0.42.35 ± 0.4210.8 ± 0.1
PALM w/o multipliers, fixed ρ\rho✗fixedper dimension65.6 ± 0.14.07 ± 0.5711.4 ± 0.3
PALM w/ fixed ρ\rho✓fixedper dimension73.5 ± 0.24.70 ± 0.384.71 ± 0.19
PALM w/ scalar constraints✓adaptive∥ct∥22=0\lVert c_t \rVert_2^2 = 061.1 ± 0.51.77 ± 0.2814.4 ± 0.1
PALM✓adaptiveper dimension74.1 ± 0.12.23 ± 0.044.65 ± 0.11

Removing the multipliers, which turns PALM into a quadratic penalty method, drops SR by 6 points and more than doubles the rollout loss. PALM with fixed ρ\rho reaches similar SR only when ρ\rho is tuned carefully. Even then, its dynamics residual is 2.1x higher than with PALM. PALM does not require tuning ρ\rho, unlike the fixed-ρ\rho variants. We swept ρ\rho for the fixed-ρ\rho variants and ρinit\rho_{\text{init}} for PALM (App. A.3 of the revised paper). Across the swept values, SR ranges from 32.9 to 73.2 with fixed ρ\rho, but varies by at most 3 points with PALM, because the adaptive schedule corrects the initial value of ρ\rho. Having one constraint per latent dimension is also better than having a single squared norm constraint: PALM w/ scalar constraints drops SR by 13 points.

Experiments on a new environment: Cube

We added experiments on the OGBench Cube environment (single-cube), with 5-D actions and 19-D proprioception. We trained the world model on the expert dataset of LeWM (Maes et al., 2026) with the same architecture as for the other tasks, with a latent dimension of 1,587. We set PALM’s tolerance floors τmin⁡m\tau^m_{\min} with the same rule as on the other tasks.

Table T2. SR (%) ↑ on Cube for each goal distance k (horizon T = k/5), and planning time ↓ for a batch of 50 start–goal pairs at k = 25. 300 start–goal pairs per k.

Plannerk = 25k = 30k = 35k = 40Time, k = 25 (min)
CEM45.7 ± 0.542.1 ± 0.736.1 ± 0.931.9 ± 0.210.48 ± 0.08
MPPI25.4 ± 1.022.8 ± 0.820.9 ± 1.321.7 ± 2.310.45 ± 0.03
PALM63.1 ± 0.650.8 ± 1.542.6 ± 1.337.2 ± 1.32.00 ± 0.05
AuthorsOpenReview ↗

To all reviewers: paper additions

We made the following changes to the revised paper:

  • Tab. 1 and Fig. 2: results over 3 seeds, and MPPI added.
  • Fig. 4, and Figs. 6 and 11 (App. A.4): MPPI added. MPPI hyperparameters in Tab. 4 (App. A.2.7).
  • Sec. 2 ("Model-based RL"): new paragraph on earlier model-based RL methods.
  • Sec. 4.3 and App. A.5.1 (Tab. 8): final dynamics residuals and stopping-criterion statistics (T4).
  • Sec. 4.4 and App. A.5.2 (Tab. 9): SS-Grad with matched objective and budget (T3).
  • Sec. 4.6: discussion of gradient-based vs. sampling-based planners.
  • Sec. 4.7 and App. A.3 (Tab. 5): ablations of PALM's components (T1) and the fixed-ρ\rho sweep.
  • App. A.2.1: extension of the constraints to a predictor with history HH.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.