- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper introduces an advantage‑aware action‑chunk MBRL framework for online planning called SWAN. The authors claim two limitations of conventional MPPI-based methods: lack of capability of modeling the temporal correlations and multimodal distributions of complete action sequences, and the misalignment between policy‑optimization objectives and planner targets. In order to deal with these, the authors combine k‑MPPI with a MFA policy to model multimodal action‑chunk distributions, and proposed Advantage‑Guided Proposal Optimization to improve the quality of candidate actions while maintaining alignment with the planner. The authors conducted evaluations across multiple benchmarks, along with simulation and real‑world deployment of a quadruped robot. The results demonstrate that SWAN reduces MPPI optimization iterations by approximately 50% while achieving state-of-the-art performance.
Critical strengths
- This paper identifies two meaningful limitations in MPPI-based model-based reinforcement learning: the inability of modeling full temporal structure and multimodality of action chunks, and the dilemma between improving policy returns and maintaining alignment with the online planner. The distinction between proposal representation and proposal optimization provides a clear motivation for the proposed framework.
- The evaluation covers 31 continuous-control tasks across DMControl, HumanoidBench, and MyoSuite, as well as the simulation and real-world deployment on quadruped. The paper also includes well-designed ablations of the action-chunk representation and APO. The reported reduction in MPPI refinement iterations and wall-clock evaluation results are quite promising.
Critical weaknesses
- Although the combination of k-MPPI, MFA-based action-chunk modeling, and APO gives promising results, the conceptual novelty of the overall framework is not convincing enough at this stage. The individual ingredients appear to be existing approaches in sampling-based MPC, structured action-sequence policies, and trust-region policy optimization. The paper should clarify which aspect of APO or the joint design is fundamentally new instead of a combination of ingredients.
- The theoretical analysis establishes nonnegative expected H-step advantage for the APO target and provides a sufficient condition for improvement over ideal planner imitation. However, these guarantees do not automatically imply improvement after projecting the target into the restricted MFA policy and optimizing the learned policy. It seems like there is a gap between the theoretical result and the practical algorithm, acquiring more detailed clarification.
Questions for authors
- As far as I know, k-MPPI and MFA are nothing new, and I believe applying them is not the major innovation here. Could you please claim your main novelty in this work? I understand APO is one, any other I missed?
- How much of the reported speedup remains after accounting for MFA sampling, policy inference, and all other planning overheads? How does the performance–latency trade-off change across different action-chunk horizons and refinement budgets?
- How sensitive is the APO algorithm to prediction errors in the learned world model?
- Could you please add more systematic factorial ablation that distinguish the individual contributions of multimodal planning, action-chunk modeling, and advantage-guided optimization?
7 comments
W2: There is a gap between the theoretical result and the practical algorithm, acquiring more detailed clarification.
A2: We would like to clarify that our original Theorem 2 already analyzes the learned policy after MFA projection and provides a sufficient condition for improvement over ideal planner imitation. The bound is derived directly from the projected distributions and the KL terms used in policy learning (Eq. (11)), independently of Theorem 1 (Sec. 4.3, Appendix C.3). Per your suggestion, we further make the connection between the two theorems explicit by accounting for the return distortion introduced by MFA approximation. We also provide projection diagnostics at four Dog-Run training checkpoints to evaluate whether the implemented projection preserves the target gain. (Please see Table 1 below).
Theoretical connection between Theorems 1 and 2. For the fixed latent state and world model considered in Sec. 4.3, define the return distortions
e_{\ast} &= \left\lvert \widehat J_H(\widetilde{\pi}^{\ast}) - \widehat J_H(\pi^{\ast}) \right\rvert,\\\\ e_P &= \left\lvert \widehat J_H(\widetilde{\pi}_P) - \widehat J_H(\pi_P) \right\rvert. \end{aligned}$$ The triangle inequality gives $$\widehat J_H(\widetilde{\pi}^{\ast}) - \widehat J_H(\widetilde{\pi}_P) \geq \widehat J_H(\pi^{\ast}) - \widehat J_H(\pi_P) - e_{\ast} - e_P.$$ The projected gain is positive if the exact target gain exceeds $e_{\ast}+e_P$. Substituting into Theorem 2 yields $$\begin{aligned} &\widehat J_H(\pi_\theta) - \widehat J_H(\pi_{\mathrm{IL}})\\\\ &\geq \frac{\widehat J_H(\pi^{\ast}) - \widehat J_H(\pi_P) - e_{\ast} - e_P}{1+\beta}\\\\ &\quad - A_{\max} \sqrt{\frac{2\left[\mathrm{KL}(\widetilde{\pi}^{\ast} \Vert \pi_\theta) + \beta\thinspace\mathrm{KL}(\widetilde{\pi}_P \Vert \pi_\theta)\right]}{1+\beta}}. \end{aligned}$$ Here $\pi_{\mathrm{IL}}=\widetilde{\pi}_P$. Under the assumptions of Theorem 2, a positive right-hand side is sufficient for improvement over ideal planner imitation. **Empirical projection diagnostics.** With all model and policy parameters frozen, we compare the exact target gain $\Delta_{\mathrm{disc}}=\widehat J_H(\pi^{\ast})-\widehat J_H(\pi_P)$ and the projected target gain $\Delta_{\mathrm{MFA}}=\widehat J_H(\widetilde{\pi}^{\ast})-\widehat J_H(\widetilde{\pi}_P)$ at four training checkpoints. Each checkpoint uses 500 states from 10 evaluation episodes and three planning repeats per state. Discrete expectations are computed exactly from the retained candidates, and MFA expectations are estimated by sampling. Table 1 shows positive estimated projected gains at all 500 states per checkpoint after averaging the planning repeats. The mean projected gain also exceeds the exact target gain, with 95% CIs for the difference above zero at every checkpoint. The additional results support preservation of the target gain through the implemented MFA projection, complementing Theorem 2's sufficient condition for improvement after policy learning. Table 1. Model-estimated target gains on Dog-Run at four training checkpoints. Gain estimates are reported as means [95% episode-bootstrap CIs]. | Env. Steps ($\times 10^6$) | $\Delta_{\mathrm{disc}}$ | $\Delta_{\mathrm{MFA}}$ | $\Delta_{\mathrm{MFA}}-\Delta_{\mathrm{disc}}$ | States with positive $\Delta_{\mathrm{MFA}}$ | |:---|---:|---:|---:|---:| | 0.2 | 0.076, [0.058,0.094] | 0.082, [0.062,0.103] | 0.006, [0.003,0.009] | 500/500 | | 0.6 | 0.252, [0.231,0.277] | 0.269, [0.246,0.296] | 0.017, [0.015,0.020] | 500/500 | | 1.2 | 0.354, [0.342,0.366] | 0.381, [0.369,0.394] | 0.027, [0.026,0.029] | 500/500 | | 2.0 | 0.233, [0.218,0.252] | 0.252, [0.234,0.274] | 0.019, [0.016,0.023] | 500/500 | Q1: Could you please claim your main novelty in this work? I understand APO is one, any other I missed? -- **A(Q1):** We would like to clarify that **k-MPPI is first proposed in this work, jointly with, to our knowledge, the first action-chunk policy that uses MFA to explicitly learn the complete $H$-step planner proposal online.** In addition to APO, **the joint design of k-MPPI and the MFA action-chunk policy is our other main methodological contribution.** The retained planner modes provide corresponding MFA learning targets, allowing the policy to learn mode probabilities and correlations across time and action dimensions for subsequent planning. The joint design addresses the proposal representation gap. As also noted by Reviewer gyCE, learning from the planner's complete chunk distribution is “a concrete distinction from first-action imitation.” Reviewer gTLb likewise highlights the combination of direct action-chunk sampling and multimodal planning as a reasonable approach to reducing online planning cost. **Please see W1 for the detailed novelty clarification.**Q2.1: How much of the reported speedup remains after accounting for MFA sampling, policy inference, and all other planning overheads?
A(Q2.1): We would like to clarify that the reported speedup over BMPC in our paper already includes MFA sampling, policy inference, and all other planning overheads. Table 1 in the manuscript reports end-to-end wall-clock evaluation time averaged over 7 DMControl tasks (10 episodes per task, ). SWAN () takes 18.12 s versus 27.90 s for BMPC (), a 35.1% reduction. We have clarified this timing protocol in Sec. 5.1 of the updated manuscript, with the changes highlighted in BLUE.
Q2.2: How does the performance–latency trade-off change across different action-chunk horizons and refinement budgets?
A(Q2.2): Per your suggestion, we evaluate the performance–latency trade-off across different planning horizons and refinement budgets on Dog-Run under environment steps.
Specifically, on Dog-Run, with planning horizon , using 4 MPPI refinement iterations () achieves a similar mean return to with lower evaluation time, while reduces evaluation time at the cost of lower return. We report end-to-end wall-clock evaluation time over 10 episodes in Tables 2 and 3 below, following Table 1 in the manuscript.
Table 2. Performance and evaluation time on Dog-Run across MPPI refinement budgets.
| Metric | |||
|---|---|---|---|
| Mean return ↑ | 694.73 | 766.67 | 762.01 |
| Eval. time (s) ↓ | 18.35 | 21.88 | 30.47 |
For the horizon comparison, we evaluate on a frozen action-chunk policy with . We use the first-action marginal for and policy rollouts with latent world model to extend candidates for . Increasing the planning horizon from 1 to 3 improves mean return at additional evaluation cost, while extending it to 5 further increases cost without improving return.
Table 3. Planning-horizon sensitivity on Dog-Run with a fixed three-step policy and .
| Metric | |||
|---|---|---|---|
| Mean return ↑ | 678.43 | 766.67 | 740.51 |
| Eval. time (s) ↓ | 18.30 | 21.88 | 28.51 |
Q3: How sensitive is the APO algorithm to prediction errors in the learned world model?
A(Q3): We would like to clarify that APO uses advantage estimates from k-MPPI trajectory evaluations, so errors in model-estimated returns can affect proposal improvement (Sec. 4.2). Per your suggestion, we examine this dependence within SWAN by freezing the encoder and dynamics model after or environment steps, while continuing to update the reward and value models, policy, and alignment multiplier.
We evaluate all three final models on the same held-out dataset pooled from 10 episodes per model (14,940 three-step windows). Then we measure per-dimension latent prediction MSE after one () and three () open-loop steps along the recorded actions, comparing predictions with future observations encoded by each model's own encoder.
Table 4 shows that earlier freezing produces higher one-step and three-step latent prediction errors and lower final return, despite continued APO and planner-alignment updates.
Table 4: Dog-Run results with and 4 MPPI refinement iterations (). All settings are evaluated after environment steps of training. Columns indicate the environment steps up to which the encoder and dynamics are updated.
| Metric | |||
|---|---|---|---|
| Final return ↑ | 428.14 | 634.42 | 766.67 |
| () ↓ | 0.544 | 0.488 | 0.340 |
| () ↓ | 1.791 | 1.513 | 1.068 |
Thank you for your constructive comments. We have addressed all the comments and suggestions from the Reviewer. We hope our responses below address your concerns. Please let us know if you have any additional concerns.
W1: The paper should clarify which aspect of APO or the joint design is fundamentally new instead of a combination of ingredients.
A1: Both the joint k-MPPI and MFA proposal representation and APO are new methodological contributions introduced in this work, rather than a direct combination of existing approaches. They are developed to address the proposal representation and optimization gaps by explicitly representing and improving the complete H-step planner proposal for efficient online planning (Secs. 1 and 4).
Novel joint proposal representation. We introduce k-MPPI for the first time and jointly develop, to our knowledge, the first action-chunk policy that uses MFA to explicitly represent and learn the complete H-step planner proposal in online MBRL. Unlike the single diagonal-Gaussian projection in MPPI control (Eq. (2)), k-MPPI maintains K policy-initialized local searches and global exploration under a fixed total sampling budget. To learn the resulting proposal, we associate retained search modes with MFA components and construct corresponding targets from branch assignments and weighted candidates. The policy learns mode probabilities and correlations across time and action dimensions, then initializes subsequent planning, addressing the proposal representation gap through this joint design (Sec. 4.1, Eqs. (4)–(6); Algorithm 1). A visualization comparing k-MPPI with standard MPPI is available on the anonymous project website linked in the abstract https://swanfastworldplanner.github.io/.
Novel proposal optimization. APO introduces a new formulation in which the current complete planner proposal is both the object of advantage-guided improvement and the reference for planner alignment during policy learning. Unlike planner imitation, APO uses advantage estimates from MPPI trajectory evaluations to improve the policy learning target. Since an improved target alone does not directly constrain the learned policy, APO trains πθ toward π∗ under an explicit trust region around πP. The formulation addresses the proposal optimization gap through proposal improvement and planner alignment, improving policy guidance for subsequent planning (Sec. 4.2, Eqs. (7)–(11)).