acceptodds
Under review as a conference paper at ICLR 2027

Fast World Planner: Advantage-Aware Action-Chunk Policy for Online Planning

14 views

Abstract

Model Predictive Path Integral (MPPI) control is central to online planning for model-based reinforcement learning (MBRL), but its iterative refinement incurs planning latency. Reducing refinement iterations often degrades performance, motivating stronger policy guidance learned from the planner. However, existing methods typically learn from the planner through action-level alignment, which does not directly capture the complete -step planner proposal. Moreover, planner imitation leaves further target improvement to online refinement. To address these limitations, we propose Fa**S**t **W**orld Pl**AN**ner (**SWAN**), an advantage-aware MPPI-based MBRL framework that explicitly represents and improves the complete -step planner proposal to improve policy guidance for efficient online planning. For proposal representation, SWAN combines k-MPPI, a multimodal MPPI extension, with a mixture of factor analyzers (MFA) action-chunk policy to capture multiple modes and correlations across time and action dimensions. For proposal optimization, we introduce Advantage-Guided Proposal Optimization (APO), which reuses MPPI evaluations to construct an improved target using advantage estimates and trains the policy toward this target while preserving planner alignment. Experiments across 31 high-dimensional continuous-control tasks from DMControl, HumanoidBench, and MyoSuite show that SWAN substantially improves control performance and sample efficiency while using 50% fewer MPPI refinement iterations than state-of-the-art planning-based MBRL methods. Finally, SWAN demonstrates zero-shot sim-to-real transfer on a physical Unitree Go2, achieving stable velocity-tracking locomotion with high-frequency control. (see our demo at [https://swanfastworldplanner.github.io](https://swanfastworldplanner.github.io)).

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Model-based Reinforcement Learning, Online Planning, Action-Chunk Policy

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tRsrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper introduces an advantage‑aware action‑chunk MBRL framework for online planning called SWAN. The authors claim two limitations of conventional MPPI-based methods: lack of capability of modeling the temporal correlations and multimodal distributions of complete action sequences, and the misalignment between policy‑optimization objectives and planner targets. In order to deal with these, the authors combine k‑MPPI with a MFA policy to model multimodal action‑chunk distributions, and proposed Advantage‑Guided Proposal Optimization to improve the quality of candidate actions while maintaining alignment with the planner. The authors conducted evaluations across multiple benchmarks, along with simulation and real‑world deployment of a quadruped robot. The results demonstrate that SWAN reduces MPPI optimization iterations by approximately 50% while achieving state-of-the-art performance.

Critical strengths
  1. This paper identifies two meaningful limitations in MPPI-based model-based reinforcement learning: the inability of modeling full temporal structure and multimodality of action chunks, and the dilemma between improving policy returns and maintaining alignment with the online planner. The distinction between proposal representation and proposal optimization provides a clear motivation for the proposed framework.
  2. The evaluation covers 31 continuous-control tasks across DMControl, HumanoidBench, and MyoSuite, as well as the simulation and real-world deployment on quadruped. The paper also includes well-designed ablations of the action-chunk representation and APO. The reported reduction in MPPI refinement iterations and wall-clock evaluation results are quite promising.
Critical weaknesses
  1. Although the combination of k-MPPI, MFA-based action-chunk modeling, and APO gives promising results, the conceptual novelty of the overall framework is not convincing enough at this stage. The individual ingredients appear to be existing approaches in sampling-based MPC, structured action-sequence policies, and trust-region policy optimization. The paper should clarify which aspect of APO or the joint design is fundamentally new instead of a combination of ingredients.
  2. The theoretical analysis establishes nonnegative expected H-step advantage for the APO target and provides a sufficient condition for improvement over ideal planner imitation. However, these guarantees do not automatically imply improvement after projecting the target into the restricted MFA policy and optimizing the learned policy. It seems like there is a gap between the theoretical result and the practical algorithm, acquiring more detailed clarification.
Questions for authors
  1. As far as I know, k-MPPI and MFA are nothing new, and I believe applying them is not the major innovation here. Could you please claim your main novelty in this work? I understand APO is one, any other I missed?
  2. How much of the reported speedup remains after accounting for MFA sampling, policy inference, and all other planning overheads? How does the performance–latency trade-off change across different action-chunk horizons and refinement budgets?
  3. How sensitive is the APO algorithm to prediction errors in the learned world model?
  4. Could you please add more systematic factorial ablation that distinguish the individual contributions of multimodal planning, action-chunk modeling, and advantage-guided optimization?
7 comments

Thank you for your constructive comments. We have addressed all the comments and suggestions from the Reviewer. We hope our responses below address your concerns. Please let us know if you have any additional concerns.

W1: The paper should clarify which aspect of APO or the joint design is fundamentally new instead of a combination of ingredients.

A1: Both the joint k-MPPI and MFA proposal representation and APO are new methodological contributions introduced in this work, rather than a direct combination of existing approaches. They are developed to address the proposal representation and optimization gaps by explicitly representing and improving the complete HH-step planner proposal for efficient online planning (Secs. 1 and 4).

Novel joint proposal representation. We introduce k-MPPI for the first time and jointly develop, to our knowledge, the first action-chunk policy that uses MFA to explicitly represent and learn the complete HH-step planner proposal in online MBRL. Unlike the single diagonal-Gaussian projection in MPPI control (Eq. (2)), k-MPPI maintains KK policy-initialized local searches and global exploration under a fixed total sampling budget. To learn the resulting proposal, we associate retained search modes with MFA components and construct corresponding targets from branch assignments and weighted candidates. The policy learns mode probabilities and correlations across time and action dimensions, then initializes subsequent planning, addressing the proposal representation gap through this joint design (Sec. 4.1, Eqs. (4)–(6); Algorithm 1). A visualization comparing k-MPPI with standard MPPI is available on the anonymous project website linked in the abstract https://swanfastworldplanner.github.io/.

Novel proposal optimization. APO introduces a new formulation in which the current complete planner proposal is both the object of advantage-guided improvement and the reference for planner alignment during policy learning. Unlike planner imitation, APO uses advantage estimates from MPPI trajectory evaluations to improve the policy learning target. Since an improved target alone does not directly constrain the learned policy, APO trains πθ\pi_\theta toward π~∗\widetilde{\pi}^* under an explicit trust region around π~P\widetilde{\pi}_P. The formulation addresses the proposal optimization gap through proposal improvement and planner alignment, improving policy guidance for subsequent planning (Sec. 4.2, Eqs. (7)–(11)).

W2: There is a gap between the theoretical result and the practical algorithm, acquiring more detailed clarification.

A2: We would like to clarify that our original Theorem 2 already analyzes the learned policy after MFA projection and provides a sufficient condition for improvement over ideal planner imitation. The bound is derived directly from the projected distributions and the KL terms used in policy learning (Eq. (11)), independently of Theorem 1 (Sec. 4.3, Appendix C.3). Per your suggestion, we further make the connection between the two theorems explicit by accounting for the return distortion introduced by MFA approximation. We also provide projection diagnostics at four Dog-Run training checkpoints to evaluate whether the implemented projection preserves the target gain. (Please see Table 1 below).

Theoretical connection between Theorems 1 and 2. For the fixed latent state and world model considered in Sec. 4.3, define the return distortions

e_{\ast} &= \left\lvert \widehat J_H(\widetilde{\pi}^{\ast}) - \widehat J_H(\pi^{\ast}) \right\rvert,\\\\ e_P &= \left\lvert \widehat J_H(\widetilde{\pi}_P) - \widehat J_H(\pi_P) \right\rvert. \end{aligned}$$ The triangle inequality gives $$\widehat J_H(\widetilde{\pi}^{\ast}) - \widehat J_H(\widetilde{\pi}_P) \geq \widehat J_H(\pi^{\ast}) - \widehat J_H(\pi_P) - e_{\ast} - e_P.$$ The projected gain is positive if the exact target gain exceeds $e_{\ast}+e_P$. Substituting into Theorem 2 yields $$\begin{aligned} &\widehat J_H(\pi_\theta) - \widehat J_H(\pi_{\mathrm{IL}})\\\\ &\geq \frac{\widehat J_H(\pi^{\ast}) - \widehat J_H(\pi_P) - e_{\ast} - e_P}{1+\beta}\\\\ &\quad - A_{\max} \sqrt{\frac{2\left[\mathrm{KL}(\widetilde{\pi}^{\ast} \Vert \pi_\theta) + \beta\thinspace\mathrm{KL}(\widetilde{\pi}_P \Vert \pi_\theta)\right]}{1+\beta}}. \end{aligned}$$ Here $\pi_{\mathrm{IL}}=\widetilde{\pi}_P$. Under the assumptions of Theorem 2, a positive right-hand side is sufficient for improvement over ideal planner imitation. **Empirical projection diagnostics.** With all model and policy parameters frozen, we compare the exact target gain $\Delta_{\mathrm{disc}}=\widehat J_H(\pi^{\ast})-\widehat J_H(\pi_P)$ and the projected target gain $\Delta_{\mathrm{MFA}}=\widehat J_H(\widetilde{\pi}^{\ast})-\widehat J_H(\widetilde{\pi}_P)$ at four training checkpoints. Each checkpoint uses 500 states from 10 evaluation episodes and three planning repeats per state. Discrete expectations are computed exactly from the retained candidates, and MFA expectations are estimated by sampling. Table 1 shows positive estimated projected gains at all 500 states per checkpoint after averaging the planning repeats. The mean projected gain also exceeds the exact target gain, with 95% CIs for the difference above zero at every checkpoint. The additional results support preservation of the target gain through the implemented MFA projection, complementing Theorem 2's sufficient condition for improvement after policy learning. Table 1. Model-estimated target gains on Dog-Run at four training checkpoints. Gain estimates are reported as means [95% episode-bootstrap CIs]. | Env. Steps ($\times 10^6$) | $\Delta_{\mathrm{disc}}$ | $\Delta_{\mathrm{MFA}}$ | $\Delta_{\mathrm{MFA}}-\Delta_{\mathrm{disc}}$ | States with positive $\Delta_{\mathrm{MFA}}$ | |:---|---:|---:|---:|---:| | 0.2 | 0.076, [0.058,0.094] | 0.082, [0.062,0.103] | 0.006, [0.003,0.009] | 500/500 | | 0.6 | 0.252, [0.231,0.277] | 0.269, [0.246,0.296] | 0.017, [0.015,0.020] | 500/500 | | 1.2 | 0.354, [0.342,0.366] | 0.381, [0.369,0.394] | 0.027, [0.026,0.029] | 500/500 | | 2.0 | 0.233, [0.218,0.252] | 0.252, [0.234,0.274] | 0.019, [0.016,0.023] | 500/500 | Q1: Could you please claim your main novelty in this work? I understand APO is one, any other I missed? -- **A(Q1):** We would like to clarify that **k-MPPI is first proposed in this work, jointly with, to our knowledge, the first action-chunk policy that uses MFA to explicitly learn the complete $H$-step planner proposal online.** In addition to APO, **the joint design of k-MPPI and the MFA action-chunk policy is our other main methodological contribution.** The retained planner modes provide corresponding MFA learning targets, allowing the policy to learn mode probabilities and correlations across time and action dimensions for subsequent planning. The joint design addresses the proposal representation gap. As also noted by Reviewer gyCE, learning from the planner's complete chunk distribution is “a concrete distinction from first-action imitation.” Reviewer gTLb likewise highlights the combination of direct action-chunk sampling and multimodal planning as a reasonable approach to reducing online planning cost. **Please see W1 for the detailed novelty clarification.**

Q2.1: How much of the reported speedup remains after accounting for MFA sampling, policy inference, and all other planning overheads?

A(Q2.1): We would like to clarify that the reported 1.54×1.54\times speedup over BMPC in our paper already includes MFA sampling, policy inference, and all other planning overheads. Table 1 in the manuscript reports end-to-end wall-clock evaluation time averaged over 7 DMControl tasks (10 episodes per task, H=3H=3). SWAN (M=4M=4) takes 18.12 s versus 27.90 s for BMPC (M=8M=8), a 35.1% reduction. We have clarified this timing protocol in Sec. 5.1 of the updated manuscript, with the changes highlighted in BLUE.

Q2.2: How does the performance–latency trade-off change across different action-chunk horizons and refinement budgets?

A(Q2.2): Per your suggestion, we evaluate the performance–latency trade-off across different planning horizons and refinement budgets on Dog-Run under 2×1062\times 10^6 environment steps.

Specifically, on Dog-Run, with planning horizon Hplan=Hpolicy=3H_{\mathrm{plan}}=H_{\mathrm{policy}}=3, using 4 MPPI refinement iterations (M=4M=4) achieves a similar mean return to M=8M=8 with lower evaluation time, while M=2M=2 reduces evaluation time at the cost of lower return. We report end-to-end wall-clock evaluation time over 10 episodes in Tables 2 and 3 below, following Table 1 in the manuscript.

Table 2. Performance and evaluation time on Dog-Run across MPPI refinement budgets.

MetricM=2M=2M=4M=4M=8M=8
Mean return ↑694.73766.67762.01
Eval. time (s) ↓18.3521.8830.47

For the horizon comparison, we evaluate on a frozen action-chunk policy with M=4M=4. We use the first-action marginal for Hplan=1H_{\mathrm{plan}}=1 and policy rollouts with latent world model to extend candidates for Hplan=5H_{\mathrm{plan}}=5. Increasing the planning horizon from 1 to 3 improves mean return at additional evaluation cost, while extending it to 5 further increases cost without improving return.

Table 3. Planning-horizon sensitivity on Dog-Run with a fixed three-step policy and M=4M=4.

MetricHplan=1H_{\mathrm{plan}}=1Hplan=3H_{\mathrm{plan}}=3Hplan=5H_{\mathrm{plan}}=5
Mean return ↑678.43766.67740.51
Eval. time (s) ↓18.3021.8828.51

Q3: How sensitive is the APO algorithm to prediction errors in the learned world model?

A(Q3): We would like to clarify that APO uses advantage estimates from k-MPPI trajectory evaluations, so errors in model-estimated returns can affect proposal improvement (Sec. 4.2). Per your suggestion, we examine this dependence within SWAN by freezing the encoder and dynamics model after 0.6×1060.6\times 10^6 or 1.2×1061.2\times 10^6 environment steps, while continuing to update the reward and value models, policy, and alignment multiplier.

We evaluate all three final models on the same held-out dataset pooled from 10 episodes per model (14,940 three-step windows). Then we measure per-dimension latent prediction MSE after one (e1e_1) and three (e3e_3) open-loop steps along the recorded actions, comparing predictions with future observations encoded by each model's own encoder.

Table 4 shows that earlier freezing produces higher one-step and three-step latent prediction errors and lower final return, despite continued APO and planner-alignment updates.

Table 4: Dog-Run results with H=3H=3 and 4 MPPI refinement iterations (M=4M=4). All settings are evaluated after 2×1062\times10^6 environment steps of training. Columns indicate the environment steps up to which the encoder and dynamics are updated.

Metric0.6×1060.6\times 10^61.2×1061.2\times 10^62×1062\times 10^6
Final return ↑428.14634.42766.67
e1e_1 (×10−3\times10^{-3}) ↓0.5440.4880.340
e3e_3 (×10−3\times10^{-3}) ↓1.7911.5131.068
Reviewer_gyCEOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
2
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

The paper proposed method SWAN to address the question that how a learned policy give the planner better initial action sequences, so that the planner needs less search at each control step. To address the question, two challenges are identified: 1. The policy does not directly learn the complete action sequences that the planner discover 2. Imitating the current planner does not explicitly improve its learning target, leaving further improvement to online search. The paper proposed Advantage-Guided Proposal Optimization (APO) reweights evaluated chunks toward higher estimated returns, then trains the policy toward this improved distribution while preserving alignment with the planner.

Critical strengths
  1. The paper addresses challenges already recognized in sequence modeling and planning, with a design tailored to learning from an online planner. Using the planner’s complete chunk distribution as the reference for policy learning is a concrete distinction from first-action imitation. The combination of multiple search branches and an MFA policy allows direct sequence sampling and tractable distribution comparisons, giving the design a clear computational motivation.

  2. The experiments demonstrate that APO provides larger gains when fewer refinement iterations are available, supporting the intended reduction in dependence on online search.

Critical weaknesses
  1. Direct sampling and tractable KL calculations justify MFA computationally, but I am less clear about when its particular structure should produce better planning candidates. Especially earlier works, including diffusion policy and learned sampling distributions for MPC, also target the similar multimodal, coordinated action sequences challenges. Can authors further clarify why proposed method's particular structure would lead to better performance?

  2. APO’s contribution beyond established value-weighted policy improvement is not sufficiently clear to me. Can authors also clarify this?

Questions for authors

Please see weaknesses.

4 comments

Thank you for your constructive comments. We have addressed all the comments and suggestions from the Reviewer. We hope our responses below address your concerns. Please let us know if you have any additional concerns.

W1: Can authors further clarify why proposed method's particular structure would lead to better performance?

A1: Yes. Our joint k-MPPI and MFA design improves policy guidance by learning the modes and correlations of the complete planner proposal (Sec. 4.1). k-MPPI preserves separate search modes, while MFA uses low-rank factors to model coordinated variations within each mode. The policy can therefore reuse high-return action combinations found by planning, avoiding the loss of structure caused by averaging modes or sampling action coordinates independently. Fig. 6(a) supports this benefit through gains over one-step and single diagonal-Gaussian chunk policies under the same planner-imitation objective and planning budget.

Although Diffusion Policy and D-MPC also represent multimodal chunks, their cited formulations learn from offline data (Sec. 2). Our planner proposal evolves with policy and world-model updates, requiring continued adaptation of the learned modes and correlations. k-MPPI's branch assignments and weighted moments directly define explicit MFA targets, so KL fitting supervises the same distributional structure used for candidate generation (Appendices B.2–B.4). As discussed in Sec. 5.1, BMPC-DP's limited gains over BMPC may therefore reflect difficulty adapting an implicit representation online (Figs. 3–5).

W2: APO’s contribution beyond established value-weighted policy improvement is not sufficiently clear.

A2: APO's contribution lies not only in improving the current complete HH-step planner proposal to provide better policy guidance for subsequent online planning, but also in preserving planner alignment during policy learning (Secs. 1 and 4.2).

Specifically, APO uses advantage estimates from MPPI trajectory evaluations to improve the learning target relative to the current planner proposal. An improved target alone does not directly constrain the learned policy relative to the planner. APO therefore trains the policy toward the improved target under an explicit trust region around π~P\widetilde{\pi}_P, formulating proposal improvement and planner alignment together to address the proposal optimization gap.

Furthermore, we compare full SWAN with its variant without the planner-alignment term (β=0\beta=0) and extend the ablations to HumanoidBench Run and MyoSuite Key-Turn-Hard. With the same k-MPPI and MFA configurations, Table 1 shows that full SWAN achieves the highest final performance on all three tasks. On Run, APO target only (β=0\beta=0) performs worse than planner imitation, whereas full SWAN improves over both. The results support improving the policy learning target while preserving planner alignment.

Table 1: Final performance with 4 MPPI refinement iterations (M=4M=4). Training budgets are 2×1062\times10^6 environment steps for Dog-Run (DMControl) and 1×1061\times10^6 for Run (HumanoidBench) and Key-Turn-Hard (MyoSuite).

Task (metric ↑)BMPC-MFASWAN (β=0\beta=0)SWAN
Dog-Run (return)688.22727.13766.67
Key-Turn-Hard (success rate)35.33%57.67%73.33%
Run (return)540.73470.06730.51
AuthorsOpenReview ↗

Looking Forward to Your Further Comments and Feedback

Dear Reviewer gyCE,

Thank you again for your thoughtful review and helpful suggestions. We would like to briefly follow up on our response. In particular, we have clarified how the joint k-MPPI–MFA design preserves multimodality and correlations across time and action dimensions, and provides explicit targets for learning the complete planner proposal. We also explained how APO improves the proposal while preserving planner alignment during policy learning, and provided additional alignment comparisons on Run and Key-Turn-Hard to support this formulation.

We would greatly appreciate your feedback on whether these clarifications and additional results address your main concerns. Please let us know if any points require further discussion. If you find that our response has addressed your concerns, we would appreciate it if you could update your assessment accordingly. Thank you again for your time and consideration.

Dear Authors,

Thank you for the follow-up and the clarifications. I will maintain my initial positive score and raise my ratings for Presentation and Soundness.

AuthorsOpenReview ↗

Thank You for Your Feedback and Support

Dear Reviewer gyCE,

Thank you for considering our responses and updating your assessments of presentation and soundness. Your feedback helped us clarify the joint design of k-MPPI and the MFA policy, along with APO’s contribution. We appreciate your constructive engagement and continued positive assessment of our work.

Reviewer_gTLbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SWAN, a planning-based reinforcement learning method that learns distributions over complete action chunks to improve online planning efficiency. SWAN combines a multimodal extension of MPPI with a mixture-of-factor-analyzers policy that represents correlations across action dimensions and time. Its Advantage-Guided Proposal Optimization (APO) procedure reweights planner candidates using their estimated returns and trains the policy toward the resulting target with an additional planner-alignment penalty. Experiments across DMControl, HumanoidBench, and MyoSuite report improved suite-average performance with fewer planning iterations, alongside a physical Go2 locomotion demonstration.

Critical strengths
  1. The paper addresses a practically relevant problem: improving policy proposals to reduce online planning cost. The combination of direct action-chunk sampling and multimodal planning is a reasonable approach, and the evaluation includes both matched refinement budgets and comparisons against baselines using twice as many iterations.

  2. The experiments cover 31 tasks across three benchmark suites and include measured evaluation time and a physical robot demonstration. The reported gains on HumanoidBench and MyoSuite, together with the reduction in evaluation time relative to BMPC, provide encouraging evidence for the method’s practical value.

Critical weaknesses
  1. The contribution of APO needs clearer distinction from temperature sharpening and existing policy optimization methods. Combining Eqs. (4) and (9), the APO target on the retained candidates is exactly a softmax of the existing MPPI return scores with an effective temperature satisfying ηeff−1=ηMPPI−1+(λc)−1\eta_{\mathrm{eff}}^{-1} = \eta_{\mathrm{MPPI}}^{-1} + (\lambda c)^{-1}, where c=sA∑h=0H−1γhc = s_A \sum_{h=0}^{H-1}\gamma^h. The additive advantage baseline cancels in normalization. This equivalence concerns target construction; the full method also includes MFA projection and planner alignment. Nevertheless, it leaves unclear what the proposed target construction contributes beyond a separately chosen, return-normalized distillation temperature. The paper should also discuss Maximum a Posteriori Policy Optimisation, which already combines exponential reweighting with supervised policy fitting and separate KL constraints.

  2. Missing related work. The authors should discuss Inference Time Policy Optimization for Offline RL with Differentiable World Models (Deb et al., 2026), Learning Off-Policy with Online Planning (Sikchi et al., 2021), and SPO: Sequential Monte Carlo Policy Optimisation (Macfarlane et al., 2024). These works are relevant to the connection between model-based planning and policy improvement; discussing their relationship to SWAN would help clarify its contributions.

  3. The adaptive alignment budget does not enforce an independently specified mismatch tolerance. Equation (12) sets the budget to an exponential moving average of the observed alignment KL and updates the multiplier using the ratio of the current KL to that budget. If the KL stabilizes at a positive constant, the budget approaches the same constant and the dual update approaches zero, irrespective of the absolute mismatch. The penalty in Eq. (11) can still encourage alignment, but the implementation does not establish the fixed-tolerance interpretation suggested by Eq. (10). Comparisons with a fixed budget or fixed penalty, together with the observed KL and multiplier trajectories, are needed to assess the proposed mechanism.

  4. The representation ablations do not fully isolate the claimed sources of improvement. Figure 6a changes both the planner and the policy representation when introducing BMPC-MFA, so its gains cannot be attributed separately to k-MPPI, multiple mixture components, or within-component correlations. Figure 11 provides useful mode-count and rank comparisons, but tests only ranks 1 and 2; a rank-zero diagonal-mixture control under the same k-MPPI planner is missing. Figure 6b does isolate APO as a complete package, but does not separate target sharpening from the adaptive alignment penalty. These component studies are also restricted to Dog-Run. Similar controls on a manipulation task and a HumanoidBench task would explain the proposed explanation for the gains.

  5. Appendix B.2 defines an ideal weighted likelihood objective, but states that the implementation instead approximates the component distributions from weighted moments. It does not specify how these moments are converted into the low-rank factor and diagonal residual variances, or how the residual variances are stabilized. This operation constructs both the planner and APO targets and therefore directly affects policy learning.

  6. The theoretical results provide limited support for improvement after projection and learning. Theorem 1 establishes improvement for the exact discrete target under a fixed learned-model objective. Theorem 2 gives a conditional bound involving the difference between the expected returns of the two MFA approximations, but Theorem 1 does not establish that this projected difference is nonnegative. The paper correctly presents the result as a sufficient condition; however, it neither bounds the relevant projection errors nor evaluates whether the sufficient condition holds experimentally. The analysis therefore leaves unresolved whether the exact target improvement survives the approximation and learning steps used by SWAN.

  7. The uncertainty and baseline reporting leave some comparisons difficult to assess. The experiments use three training seeds, and Appendix G defines final performance using the last evaluation checkpoint without terminal-window averaging. The sample-efficiency threshold is then determined by this final value for SWAN. Although the threshold is shared across methods, its sensitivity to checkpoint noise is not evaluated. This matters particularly for smaller differences, such as the approximately 1.7% DMControl suite improvement over PO-MPC. Terminal-window results and uncertainty on performance differences and threshold-crossing times would strengthen the evidence. Separately, Table 10 reports zero final success for FPO on all ten MyoSuite tasks, while Appendix E.2 specifies default configurations for DMControl without comparable implementation details for the other suites. These configurations and baseline validation should be documented.

Questions for authors

Please also check Weaknesses.

  1. Can you compare APO with planner imitation using a separately tuned target temperature while keeping the planner and MFA representation fixed? Within APO, how much of the gain comes from target sharpening versus the adaptive alignment penalty?

  2. Does the EMA-based budget maintain a small absolute alignment KL in practice? Please report the KL and multiplier trajectories and compare against a fixed-budget or fixed-penalty alternative.

12 comments

Thank you for your constructive comments. We have addressed all the comments and suggestions from the Reviewer. We hope our responses below address your concerns. If our clarifications and additional results address your concerns, we would appreciate it if you could consider increasing your score. Please let us know if any concerns remain.

W1.1 & Q1: It's unclear what the proposed target construction contributes beyond temperature-sharpening. Comparison with planner imitation using a separately tuned target temperature and the contribution of adaptive alignment.

A1.1: Please note that the "temperature-sharpening form", obtained by algebraically reorganizing Eq. (9) with Eq. (4) on fixed retained candidates, is only the learning target in our objective in Eq. (11), and can not achieve our goal of improving the complete planner proposal while preserving planner alignment during policy learning (Secs. 1 and 4.2). Specifically, APO derives Eq. (9) by maximizing normalized advantage under a KL constraint relative to the current planner proposal (Eq. (7)). However, this constraint only applies to π∗\pi^{\ast} in Eq. (9) rather than the policy πθ\pi_\theta. To further preserve planner alignment, APO trains πθ\pi_\theta toward π~∗\widetilde{\pi}^{\ast} under a forward-KL trust region around π~P\widetilde{\pi}_P (Eq. (10)).

Per your suggestion, we keep the k-MPPI planner and MFA representation fixed and compare SWAN with planner imitation using separately tuned target temperatures and with "APO target only". The distillation targets use the normalized advantage Aˉ\bar A from Sec. 4.2,

πD(τi∣zt)=softmax⁡i(Aˉ(τi)TD).\pi_D(\tau_i\mid z_t)=\operatorname{softmax}_i\left(\frac{\bar{A}(\tau_i)}{T_D}\right).

Here, TDT_D controls the distillation target, while the MPPI temperature remains unchanged. Table 1 shows that SWAN achieves the highest final return and lowest T80T_{80}. "APO target only" improves return by 38.91 over BMPC-MFA, and retaining planner alignment provides a further gain of 39.54. The results support APO's complete formulation for addressing the proposal optimization gap by improving the policy learning target while preserving planner alignment (Sec. 1 and Sec. 4.2).

Furthermore, the alignment comparisons on Run and Key-Turn-Hard in Table 5 in W4.4 also favor full SWAN, supporting APO's formulation of proposal improvement while preserving planner alignment.

Table 1. Dog-Run comparison under 2×1062\times 10^6 environment steps with 4 MPPI refinement iterations (M=4M=4). The TDT_D groups use fixed return-normalized distillation temperatures, the same MFA projection, and β=0\beta=0. SWAN (β=0\beta=0) denotes APO target only. T80T_{80} uses 80% of SWAN's final return (766.67) as the shared threshold.

MetricSWANBMPC-MFATD=0.05T_D=0.05TD=0.1T_D=0.1TD=0.2T_D=0.2SWAN (β=0\beta=0)
Final return ↑766.67688.22711.53727.17700.14727.13
T80T_{80} (×106\times 10^6 env. steps) ↓1.001.201.201.501.201.10

W1.2: The paper should also discuss MPO [1], which already combines exponential reweighting with supervised policy fitting and separate KL constraints.

A1.2: MPO [1] is a model-free, off-policy RL algorithm. In contrast, SWAN is a planning-based MBRL framework for efficient online planning. Within SWAN, our Advantage-guided Proposal Optimization (APO) learns improved action-chunk proposals for subsequent planning under explicit planner alignment. We added this comparison to the related work in the updated manuscript, with the changes highlighted in BLUE, and detail the distinctions below.

  1. Advantage-guided proposal improvement. Accordingly, APO in our SWAN uses advantage estimates from existing MPPI trajectory evaluations to improve the current complete HH-step planner proposal, defined by the retained candidates and their weights (Eqs. (4), (7)–(9)). APO then trains the policy toward the target’s MFA approximation to improve subsequent planning (Sec. 4.2). However, though MPO also uses exponential reweighting and policy fitting, it aims to improve an action policy for control and constructs its target from the old policy using action-value estimates.

  2. Planner alignment. Furthermore, APO trains πθ\pi_\theta under an explicit forward-KL trust region around π~P\widetilde{\pi}_P, the MFA approximation of the current planner proposal, to preserve planner alignment (Eq. (10)). MPO’s M-step instead constrains policy changes relative to the old policy.

W2: Discuss LOOP [2], SPO [3], ITPO [4].

A2: Per your suggestion, we discuss LOOP [2], SPO [3], and ITPO [4] below. We have also added these works to the related work in the updated manuscript, with the changes highlighted in BLUE.

SWAN uses a learned latent world model and aims to improve policy guidance for efficient online planning (Sec. 3 and Sec. 4). In contrast, LOOP and ITPO model dynamics in the environment state space, while SPO is evaluated with exact environment models.

Furthermore, LOOP identifies actor divergence between its lookahead policy and actor, corresponding to the policy-planner mismatch discussed in our paper (Sec. 1 and Sec. 3). To reduce this mismatch, LOOP uses ARC to constrain the planner’s action-sequence distribution relative to an actor-derived prior, while its online actor uses SAC updates. In contrast, SWAN represents the complete HH-step planner proposal with an MFA action-chunk policy, and uses APO to improve the learning target using advantage estimates and train the policy toward the improved target under an explicit planner-alignment constraint (Sec. 1 and Sec. 4.2).

Within SPO’s EM framework, advantage-weighted SMC search estimates an E-step target under a KL constraint relative to the old policy. The M-step fits the policy to the first-action marginal, with an additional KL constraint relative to the old policy. However, SWAN applies advantage estimates from MPPI trajectory evaluations to improve the current complete HH-step planner proposal under a KL constraint. SWAN then trains the MFA action-chunk policy toward the improved target under an explicit trust region around the planner proposal to preserve planner alignment (Sec. 4.2).

As for ITPO, it starts from a policy and differentiable world model trained on offline data, whereas SWAN learns its world model and policy through online interaction (Alg. 2). At inference time, ITPO updates policy parameters by differentiating predicted returns through model rollouts before action execution, while SWAN uses k-MPPI for online action-sequence optimization. Specifically, the APO in SWAN uses advantage estimates to improve the complete HH-step planner proposal under a KL trust region and trains the action-chunk policy toward the improved target (Sec. 4.1–4.2).

W3.1: The adaptive alignment budget does not enforce an independently specified mismatch tolerance.

A3.1: We would like to clarify that Eq. (12) selects an adaptive alignment budget for Eq. (10), rather than enforcing an independently prescribed absolute mismatch tolerance.

Specifically, Eq. (10) formulates policy learning for a given budget, and Eq. (11) gives the corresponding Lagrangian. Since alignment KL scales can differ across tasks, a fixed absolute budget may require task-specific tuning. We therefore use the EMA of the observed alignment KL as the current budget and normalize the multiplier update to reduce sensitivity to the absolute KL scale (Sec. 4.2). The budget is held fixed during each policy and dual update but adapts over training. A stable positive KL can yield a vanishing multiplier update, while a positive β\beta retains the alignment penalty in Eq. (11). We have clarified the adaptive budget choice in the updated manuscript, with the changes highlighted in BLUE.

W3.2 & Q2.2: Comparisons with a fixed budget or fixed penalty, together with the observed KL and multiplier trajectories, are needed to assess the proposed mechanism.

A3.2: Per your suggestion, we compare our SWAN against SWAN with fixed-penalty APO on Dog-Run. Table 2 reports control performance and sample efficiency, while Table 3 reports the per-dim alignment KL and multiplier trajectories.

As shown in Tables 2 and 3, SWAN achieves a higher final return and lower T80T_{80} than β=0\beta=0, while maintaining lower per-dim alignment KL. A smaller fixed coefficient allows larger mismatch, whereas increasing the fixed coefficient from 0.8 to 3.2 further reduces KL but lowers final return. SWAN achieves a higher final return and the same T80T_{80} as the best tested fixed coefficient, with the adaptive β\beta remaining near 0.8. The comparisons support the effectiveness of the adaptive update in balancing proposal improvement and planner alignment.

Table 2. Comparison of final return and sample efficiency across SWAN and SWAN with fixed-penalty APO on Dog-Run with 2×1062\times 10^6 environment steps and 4 MPPI refinement iterations (M=4M=4). T80T_{80} uses 80% of SWAN's final return (766.67) as the shared threshold.

MetricSWANβ=0\beta=0β=0.2\beta=0.2β=0.8\beta=0.8β=3.2\beta=3.2
Final return ↑766.67727.13735.92750.26712.69
T80T_{80} (×106\times 10^6 env. steps) ↓1.01.11.01.01.0

Table 3. Alignment KL per action-chunk dimension and multiplier trajectories from the same Dog-Run experiments as Table 2. The adaptive β\beta corresponds to SWAN.

Env. steps (×106\times 10^6)Adaptive β\betaSWAN KLβ=0\beta=0 KLβ=0.2\beta=0.2 KLβ=0.8\beta=0.8 KLβ=3.2\beta=3.2 KL
0.0060.8320.1140.5030.1750.1150.083
0.10.7870.1430.1970.1720.1470.129
0.20.8010.1460.1820.1660.1420.123
0.60.8100.1490.1670.1630.1510.121
1.20.8240.1560.1660.1640.1500.122
2.00.7960.1470.1620.1600.1470.123

W4.1: Figure 6a changes both the planner and the policy representation when introducing BMPC-MFA, so its gains cannot be attributed separately

A4.1: We would like to clarify that Figure 6a aims to evaluate k-MPPI and MFA as a joint representation of the complete HH-step planner proposal, rather than attributing gains to each component independently.

Our representation gap concerns learning the complete action-chunk distribution, not merely extending the output horizon. A single diagonal Gaussian cannot capture multiple modes or correlations across time and action dimensions. We therefore introduce k-MPPI and MFA together to preserve search modes and model their corresponding action-chunk distributions (Sec. 4.1). Accordingly, Figure 6a shows that BMPC-MFA outperforms BMPC and BMPC-Diag-G on Dog-Run, supporting the proposed joint representation over one-step policy learning and single-diagonal-Gaussian chunk modeling (Sec. 5.1).

W4.2: Figure 11 in Appendix E.4 misses a rank-zero diagonal-mixture control under the same k-MPPI planner.

A4.2: We would like to clarify that Figure 11 aims to examine mode count and factor rank within our joint proposal representation and already supports our choice of K=2,r=1K=2,r=1. Per your suggestion, we further evaluate rank-zero controls under the same k-MPPI planner with K∈{1,2}K\in\{1,2\}, and examine within-component correlations. We have updated Figure 11 in Appendix E.4 of the manuscript, with the changes highlighted in BLUE.

Specifically, Table 4 compares the diagonal-covariance (r=0r=0) and r=1r=1 variants at each KK, with BMPC as a reference. As shown in Table 4, increasing rr from 0 to 1 improves final return and sample efficiency (smaller T80T_{80}) at both mode counts, further supporting the benefit of within-component correlations across time and action dimensions. Increasing KK from 1 to 2 also improves both metrics at each rank, supporting multiple modes in the complete HH-step planner proposal. These additional controls strengthen the evidence for multiple modes and within-component correlations within our joint proposal representation.

Table 4: Comparison of final return and sample efficiency across BMPC and BMPC-MFA with different KK and rr under planner imitation on Dog-Run with 2×1062\times 10^6 environment steps and 8 MPPI refinement iterations (M=8M=8). T80T_{80} uses 80% of SWAN's final return (766.67) as the shared threshold.

MetricBMPCK=1,r=0K=1,r=0K=1,r=1K=1,r=1K=2,r=0K=2,r=0K=2,r=1K=2,r=1
Final return ↑684.89657.95739.56696.49766.61
T80T_{80} (×106\times 10^6 env. steps) ↓1.201.401.201.100.9

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.