- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PALM, a first-order augmented Lagrangian planner for learned latent dynamics. The method jointly optimizes intermediate latent states and actions, enforcing dynamics consistency through vector-valued multipliers and adaptive penalty coefficients, while using Adam to approximately solve the inner subproblems. Using the same world model, the experiments compare PALM with single-shooting, multiple-shooting, and sampling-based planners. The results show that PALM substantially outperforms the evaluated gradient-based planners on several challenging navigation tasks and outperforms CEM under certain computational budgets.
Critical strengths
- The paper presents a well-motivated planning approach with clear methodological distinctions. PALM combines first-order optimization with an augmented Lagrangian treatment of vector-valued dynamics residuals, distinguishing it from the second-order LatCo implementation evaluated in this paper and the pure-penalty approach used by GRASP. Although the underlying optimization tools are established, their effective adaptation to high-dimensional learned latent spaces has both research and practical value. The paper also clearly positions the approach relative to prior work.
- The experiments demonstrate practically meaningful performance improvements. All planners share the same world model, planning instances, and horizons, and LatCo is adapted to the same 1,578-dimensional latent state. On the longest Wall crossing tasks, PALM achieves an 82.3% success rate, compared with 31.0% for SS-Grad and 22.7% for GRASP. The success-versus-time curves further demonstrate its value under limited planning budgets.
Critical weaknesses
The individual contributions of the key design choices have not been established through controlled ablations. The comparisons with LatCo and GRASP support the effectiveness of PALM as a complete approach. However, these methods differ simultaneously in their constraint formulations, optimizers, gradient handling, and update settings, making it difficult to determine how much improvement comes from multiplier updates and adaptive penalties, respectively. Since augmented Lagrangian constraint handling is an important aspect of the proposed method, comparing the full PALM with variants that disable multiplier updates or use fixed penalties, while keeping other settings unchanged, would help identify the sources of the performance gains.
Questions for authors
- Which key design choices account for PALM’s performance gains? Could you compare the full PALM with variants that disable multiplier updates or use fixed penalty coefficients, while keeping other settings unchanged? Reporting success rates and dynamics residuals under the same computational budget would help clarify the individual contributions of multiplier updates and adaptive penalties.
- How do the authors interpret the trade-off between computational efficiency and success rate when comparing PALM with CEM? The paper demonstrates PALM’s advantages under certain time budgets and CEM’s higher final success rates on some tasks. What task characteristics or computational conditions favor choosing PALM?
- What empirical observations do the authors have regarding PALM’s dependence on world-model accuracy? The paper discusses model error through its tolerance settings and a failure case. Building on this analysis, have the authors observed whether PALM is more or less prone to exploiting model errors than other planners? Such a discussion would help clarify the conditions under which the method is effective.
- Which design choices can be carried over directly to broader planning settings, and which might require modification? For example, would the current initialization, constraint-update rules, and stopping criteria remain suitable for closed-loop MPC or tasks involving intermediate-state costs?
2 comments
Answer to reviewer 6CxE 2/2
Which design choices can be carried over directly to broader planning settings, and which might require modification? For example, would the current initialization, constraint-update rules, and stopping criteria remain suitable for closed-loop MPC or tasks involving intermediate-state costs?
- In principle, all design choices can be carried over to closed-loop MPC. We added closed-loop MPC experiments on PointMaze rest goals (Table T6) with the constraint-update rules and stopping criteria unchanged. We used the same initialization for the first planning step, and at each replanning step, we warm-started the nodes and actions from the previous solution, shifted by one step. We see that with MPC, PALM’s SR increases at all goal distances. We also ran MPC with SS-Grad, to show that PALM's performance does not come from replanning alone. Due to limited compute, we ran these experiments on PointMaze rest goals only. We picked this task because PALM performs worst there in open loop relative to sampling-based planners, and its SR drops the most as the goal gets further. We will include MPC results on all environments in the final version.
Table T6. SR (%) on PointMaze rest goals, open loop (T = k/5) vs. closed-loop MPC (T = 5, 15 replanning steps). 50 episodes per k.
| Planner | k = 25 | k = 30 | k = 35 | k = 40 |
|---|---|---|---|---|
| Open loop | ||||
| SS-Grad | 51.3 ± 8.4 | 30.7 ± 0.9 | 20.0 ± 5.7 | 21.3 ± 4.1 |
| PALM | 74.7 ± 4.1 | 58.0 ± 5.9 | 55.3 ± 1.9 | 40.0 ± 7.1 |
| MPC, warm start | ||||
| SS-Grad | 54.0 ± 5.9 | 47.3 ± 3.4 | 48.0 ± 5.7 | 46.0 ± 1.6 |
| PALM | 89.3 ± 3.4 | 80.7 ± 5.2 | 74.7 ± 5.7 | 70.0 ± 3.3 |
- Intermediate-state costs fit the multiple shooting formulation directly: the costs would be added to the objective of Eq. 2, and the constraint update rules and stopping criteria stay the same. What might require tuning is the weight of these costs relative to the penalty coefficients .
We hope these answers address the reviewer's concerns, and we are happy to answer any further questions during the discussion period.
Answer to reviewer 6CxE 1/2
We thank the reviewer for their positive review, and answer their questions and concerns below.
Thank you for raising this point! We agree, and we added an ablation of PALM (see our answer to the first question below).
We added these ablations in Table T1 (comment “To all reviewers”). We compare PALM with variants that remove the multiplier updates, fix the penalty coefficients, or both, under the same computational budget and report SR and dynamics residuals.
We believe that because CEM samples a pool of candidates to evaluate at every iteration, given enough time it is able to explore the non-convex space better and reach higher final success. PALM on the other hand optimizes a single candidate (actions and nodes), which makes iterating faster and allows it to reach competitive success faster. We expect PALM to be preferable in two settings:
We measured how planners exploit model errors by comparing the latent goal distance reached by their final actions in the world model and in the environment (Table T5). Sampling-based planners exploit model errors least on Wall and PointMaze, consistent with the observation that gradient-based optimization is more prone to finding actions where the model is inaccurate [1]. Among gradient-based planners, PALM has the smallest gap on all tasks except PointMaze passthrough, and the smallest on average. One explanation is that PALM calls the model only one step ahead and sets tolerance floors based on the model prediction error, preventing optimization beyond model accuracy.
PALM depends on the one-step accuracy of the model: it stops when the residuals reach the model’s one-step error, so small errors at each step can add up over the rollout. This is the case on PointMaze, where the forward rollout drifts from the shooting nodes (Table T4, comment to reviewer RG5x). PALM is therefore most effective when the one-step prediction error of the model is small. In closed-loop MPC, replanning corrects this drift (Table T6).
Table T5. Gap between the latent goal distance reached in the environment and in the world model (×10⁻³, lower is better). Median over episodes. Bold: best among gradient-based planners.
[1] Parthasarathy et al., Closing the Train-Test Gap in World Models for Gradient-Based Planning, arXiv, 2025.